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The purpose of this review was to examine research 
designs in studying knowledge utilization. The results are based on 
32 studies of knowledge utilization, and the report describes the 
various types of research designs and their strengths and weaknesses. 
Survey research methods are appropriate for dealing with either of 
two aspects of a qualitative research topic: issues of frequency, and 
issues of perception and attitude. Beyond these issues, survey 
strategies have limited applications in qualitative research. The 
case study was the most common research strategy used in these 32 
studies. Although multiple-case studies are able to deal with the 
entire knowledge utilization process in an explanatory manner, they 
are not good tools for establishing the frequency or extensiveness of 
a particular phenomenon. Mixed designs are those in which a single 
investigation relies on both the survey and ca?e study strategies. In 
general, these designs offer advantages that neither the survey alone 
nor the case study alone can provide. Four types of project 
management issues can affect the research design: project staff 
skills, project organization, resources available, and reporting 
requirements. Potential implications of these findings are discussed. 

(bw) 



*********************************** 

^ Reproductions supplied by EDRS are the best that can be made ^ 

* ^ from the original document. * 

****it************** ************************* *************************** 



\JS. DEPARTMENT OF EDt^CATiON 
MATJDMAL JMST)tt;T£ OF GDUCATJDM 
EDUCATlOrjAL RESOURCES iNfORMATtON 

centCr <mo 

^ This docuiTr«iit J>as l>ocn rcPrufuccd 4^ 
rcCnveiJ tiOm Ihy POfSort or Orgsnv^tKyn 

MinOr dungm have t>ecn maifu to miffrov+i 

' mcnl <)t> not i^ec^^sunl/ rePnivnt olficui « lE 

fNj 
O 
UJ 



DESIGN ISSUES IN QUALITATIVE RESEARCH: 
THE CASE OF KNOWLEDGE UTILIZATION STUDIES 

r 

Robert K. Yin 
Margaret K. Gwaltney 



August 1982 



Abt Associates, Inc, 

J Washington, D.C. 

\ 
\ 
o 



2 



Hi 



PREFACE 

This r«port in the final product from a grant awarded by the National 
Institute of Education in 1981 (MI£ Grant Mo* G*81-0016)* 1)ie purpose of 
the grant was to examine research designa in studying knowledge utilize- 
tion» and the project was conducted over an 18-nonth period* 

Ilia report is based on an exhaustive review of prior atudies of 
knowledge utilization* Initially » 63 studies were Identified as a result 
of e search proceas covering existing literature and queries of active 
Inveatigators* Of these 63 atudles» however, only half (or 32) were 
eventually found to aatisfy the definitional criteria for knowledge 
utilization* The other half (or 31 atudies) examined some other topic* 
related to but not concerning knowledge utilization directly* 1)ie primary 
results of this project are therefore baaed on the 32 studies of knowledge 
utilization, and the report describes the various types of research 
designs and their strengths and weaknesses* 

This approach to our study differs from that originally proposed* 
Initially, wc had Intended to do an intenaive analyais of 16 atudies* 
Due to budget cutbacks*' however* this figure had to be reduced to about 
12* tater* as our own preliminary work progressed* we realized that a 
QK>re superficial coverage, but of a larger number of atudies* night pro* 
vide a firmer foundation for future research and action* Thus, our own 
project was affected by t'he types of fa£tora discussed swre thoroughly 
in Chapter VI. 

The lessons about research designs* drawn from our study* appear 
to be applicable to the pro'olen ot conducting qualitative research more 
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broadly* Bftiic«» ve h«ve cboaeo a «ore Seneral title for this report* 
What the report pr«eent6» for the first tlae to our knowledge* is s 
description «id conpsrison smong the different types of cesesrch designs 
that sre poscible for doing qualitstive research* Although the 
generslizstion of our findings from knowledge utilis&tion to qualitative 
resesrch loore generally requires fotmal corroborstoty evidence (as 
suggested in Chapter VII) » the major design issues appear to hsve a 
surface applicability to numerous qualitative research topics* and not 
Just those limited to knowledge utilisation* 

The data collected for our study involved a review of pertinent 
documents-»e*g* * research proposals and final reportF«as well as 
intensive discussions with several investigators who shared their 
insigjhts with us* These investigators were: Judith tarsen (Anerican 
Institutes for Research* Palo Alto); Peggie Campeau (American Institutes 
for Research* Palo Alto); Everett Rogers (Stanford University); Allen 
Parker (Center for Technology and Society* Boston); William Firestone 
(Research for Better Schools* Philadelphia); and David Crandall (The 
Network* Andover* Mass*)* Each of these investigators described In 
detsil the research design i&sues that arose In their own knowledge 
utilization studies* In addition* our project benefited from the advice 
and encouragement of an informally empanelled group of advisers* including 
Senta Raisen (National Academy of Sciences* Washington* D*C*); Karl Weick 
(Cornell University); irwln Feller (Pennsylvania State University); 
Judith Larsen (American Institutes for Research* Palo Alto); Robert 
Harriott (Cambridge* Mass.); and Lae Sechrest (University of Michigan)* 

A draft of the present report was reviewed by the above-mentioned 
advisers as well as by Judith Agard (M>t Associates* luc*)* and our KIE 
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project of£lc«r« ftolf Lchning* Dr* Lehmlng was also helpful througjhout 
urll«r eteges of the project^ and deserves epeclel aentlon es e 
particularly enll|^tened and supportive technical aponsor* Matural^^ 
neither he nor any of the others wntloned are responsible for the 
findings and coonclualons of our report* 

The findings from this study have also been presented in other 
torms* Presentations have been wide at the 1981 and 1982 meetings of 
the Dissemination Research Group» as part ot the annual meetings of the 
American Educational Research Association (AERA)* Another oral present 
tatlon la tentatively planned for the 1983 AERA meetings^ and an 
abbreviated version of this report vlll be prepared for Journal publl** 
cation* We have found these oppov't unities to Interact with other 
investigators^ during the course of a project^ to be most suggestive^ in 
our own vork» ve highly reconmend this type of activity as an adjunct to 
all research projects* 

Questions about this report and related work may be addressed to 
either author. Robert K. Yin la presently located at The Case Study 
Institute, Inc., 1730 K Street, N.W., Washington, D.C. 20006 (202-296-6595); 
and Kargaret K. Gvaltney Is located at Abt Associates* Inc., in Its new 
offices at 1055 Thomas Jefferson Street, Washington, D.C. 20007 
(202-466-4343). 
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I. THE FUNCTION OF RESEARCH DESIGNS 

Every type of empiricsl resesrch has an tmplicit* if not explicit* 
tMcsrch de«ign« In the wst elementsry tense* the design is the 
logical sequence connecting the evidence"i«e« » the empirical data** 
with the conclusions^ More apecifically» research tries to satisfy most 
of the following design requireinents (Kldder» 1981» pp« 7-8): 

Construct validity" t^c^ * the establishinenn of the appro- 
priate operational measures for the concepts being 
Studied i 

- Internal validity" i*e^ , the establishment of a causal 
relationship hereby certain conditions are shown to lead 
to or cause other conditions to occur; 

- External validity" i^e*» the establishment of the domain 
to ^ch the atudy*s findings can be generalized*; and 

- Reliability" i.c.> the demonstration that the findings are 
replicable^ 

These requirements will prevail in most social acience research* 
even though different disciplines may use different labels £or them« 
However* under certain conditions* one or more of these requirements may 
not be relevant^ For instance* some empirical studies may be descriptive 
in nature* Ib auch cases the concern Jver internal validity appro- 
priately might be ig.u)red« As another example* other atudies may focus 
on a narrow aet of events or programs* as in a program evaluation* and 
might therefore not be greatly concerned with :;he external validity 

ErJc 8 
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requircwnt. Such uceptloni iiotvlthstndliis, tfae«e four requirments 
tmerally represent the full range ot co&ceru tbrdugh which rMeerch 
deelgns dictate the d«t« to 1>e collected in «ny given etudy. 

Keeeerch Deelgns In Different Types of Sociel Science Research 

Traditionally, research designs bsve been docunented most formally 
in experinentsl «nd quasi-cxpertokentsl resesrch. Experlaents in the 
blologicsl and paychological sciences bsve followed s vell-*speclfled 
set of procedures for determining such conditions ss the sssignment of 
subjects to different "groups/' the presentstlon of different stimuli 
or experimentsl conditions, and the recording of vsrious responses (e.g.. 
Fisher, 1937; Cochran and Cox, 1957; and Sldowskl, 1966). A some^at 
independent strand of Iniquity has documented experimental designs in 
situations vhere only s small number of subjects is svsllable*"i.e* i 
"snall^n" designs (e.g.» Hersen and Barlow* 1976; cad Kratochwlll » 1976). 
Such designs must depend upon conditions other than comparisons among 
groups* because the number of aubjects la too email to form auch groups. 
Among the prominent small^n techniques are time-aeries designs* the use 
of repested trlsls, and most Important, replication p'^r'^^'^ures. 

In quasi-experimental resesrch, the investigator has less control 
in msnipulstlng the cxperlmentsl conditions. Under such circumstances, 
yet other resesrch designs are needed* mainly to deal with the internal 
validity requirement. These qua&l-experlmental resesrch designs slso 
bsve become well-known (e.g.* Csmpbell and Stanley, 1963; and Cook and 
Caapbell, 1979), both within acadeffllc research «od in the policy arena, 
trtiere program evaluations have been' frequent types ot studies. 
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In other aoclal science rMe«rch*<^.g. » in hiatory* economlc&t 
Aoclology» and political science— reaearch designs have been leas veil 
fonialiced» although the Implicit existence of aome design* as a 
logical w>del» is unavoidable* A recent developtsent » for instance* has 
been the emergence of a aeparate set of research d^itigns intended for 
use in aurvey research » in which the conditions for conducting research 
are different from those in experiniental or quasi<^xper:fiiental science. 
Thus^ the fourth edition of the popular aocial science textbook on 
ttethodological issues — Selltis et al/s Research Methods in Social 
Relations " *oov contains a separate chapter on the sost common designs 
in survey research (Kldder» 1981)* 

Overall* research investigators have become more sensitive to the 
need for specifying the research designs that underlie each individual 
atudy* If the basic logical model of the study Is flawed* the results 
of the entire study cannot be accepted. Such a problem will remain even 
if the study has otherwise been conducted in an exemplary manner* For 
example* the use of the most precise data collection measures and pro^ 
cedures will nevertheless produce an unsatisfactory study if the research 
design fails to establish a colid link between the evidence and a study* s 
conclusions* 

In spite of this increased aensitivity* however* large gaps in 
research design development and documentation still remain. Most text^ 
books only provide an enumeration of a few standard research designs* 
vainly drawn from the experimental and quasi<^xperlmental literatures* 
even though aooa of these texts have been intended for use in Social 
adence research In general (e.g.* Miller* 1970; and Meale and tiebert* 
1980) or In evaluation research (e.g. tCkossi* Freeman* and Wright* 
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1979)* r#ir textbooks ovan dtfint the $»ntTic dturacterlitlct of 
ttwarcfa d«ilgu, vhicfa vould a«cm to be a Minimal £lrat itap in pro- 
wting better docuMotatlon for deilgos 1b ell of the aocl&l acltnces^ 
imd not Just experlmenti end queel*«rperi«entB* One of the fev excep- 
tlos^e le the textbook by Nachmlas end HachiikleB (19.81), which notes that 
(pp. 77-78); 



A reseerch design Is the program thet guides the lovestl* 
getor in the process ^f collecting* analyzing, end 
interpreting observetlons* It Is e logical model of proof 
that ellotfs the researcher to drew inferences concemik^ 
ceusel reletlons among the verlebles iii:der Investigation. 
The reseerch design elso defines the domain of generellza- 
blllty» thet ls» Whether the obtelned Interpretations can 
be generalized to a larger population or to different 
situations. 



Xn generelt these euthors elso note thet the reseerch design Is the 
*1)lueprint" of reseerch thet enebles en Investigator to Identify solu- 
tions to four problems; Whom to study* what to observe, vhen to make 
observations* end how to collect the deta (see also Phllllber* Schwab, 
and SlosSt 1980» for a slmller list). In the field of eveluetlon research, 
enother exception bes been the vork of Fltz-Glbbon nnd Morris (1978)* who 
describe the generel properties of reseerch designs In the following 
menner (p. 9)£ 



A design is e plen lAxich dictates when end from Whom measure^ 
vents will be gathered during the course of en evaluation* 
The first end obvious reason for using a design Is to ensure 
e well-orgenlzed eveluetlon study: ell the right people will 
teke pert In the eveluetlon et the rlg^t times* A design, 
however t eccomplishes for the eveluator aome thing more useful 
then Just keeping deU collection on echedule* A design is 
most basically a wey of gathering comparative Information so 
that the results from the progf'em being evalueted can be 
pieced vithln e context for Judgment of their size end worth* 

> 
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Mane of thew t«Mral guidelineat however, have been eufficient 
to prottote the development of reaeerch designs for investigating topica 
outside of txperinental and quasi-experlnental aettlngs* 

Research Designs for Qualitative Research 

This gap ia especially evident in relation to a vhole het of topics 
that have come to fall under the label ''qualitative research/' For the 
purpoaes of the present report* auch research nay be considered to 
reflect the following conditions: 

An investigation involving a phenoicenon»e«g« , decision* 

iiiaklng»that has 'no clear boundaries ; 
- An investigation in which a specific unit of analysis 

is difficult to identify; 
<^ An investigation where the unit of analysis, though 

difficult to identify, is nevertheleas likely to differ 

substantially from the unit of oata collection ; and 
* An iDvestigation in which aultiple data collection 

strategies are likely to be used in converging over the 

establishment of a specific fact* event, or causal 

explanation* 

For each of these four conditions » a contrast between the traditional 
experimental investigation and the qualitative rese^tch Investigation 
may be made« 

Firatt the experimental aituation generally involves the identifi* 
cation of a specific aet of target variables, with the role of contextual 
variables held to a minimum* Typically* the classical experiment 
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linrMtlgfttM • ftinsle or mall nuaber of TarlablM uhlle holding all 
bthar varlablts coiiit«nt» by aattbllahlihs ftxpertnental and control 
cooditlona* * Xn qualitative raaaarch» this procadure is not uaually 
tfaslrad» bacauaa the phenomenon being studied cannot be divorced clearly 
froB ita contextp and in fact the boundary between phenomenon and con-^ 
text aay not be easily Identiifled (Tln» 1982b)* Such a lack of explicit 
boundariea ii one of the conditions that precludes the use of traditional 
experimental or quasi-expertniental research desigiia* 

Second » the unit of analysis nay b^ difficult to defiue in quali- 
tative research* Xn expertniental research in psychology* in contrast* 
specific types of hu&an subjects and their measured responses are ttore 
readily Identified as the units of analysis* These types of units are 
readily susceptible to the necessary sampling techniques that allow 
the use of atatistical inference* In qualitative research* the unit of 
analysis is frequently one vhich* though aeendngly atralghtforward 
at the outset* upon further investigation has a poorly articulated 
definition* For exanrple* program evaluations often assume a clear 
definition of the intervention* or **program*** Yet* on later discovery* 
the program may defy any operational characterisation (e*g** Chartres 
and Pellagrin* 19 ; and Yin* 1978)* 

Third* the unit of analysis in experiments also happens to coincide 
with the unit of data collection* Thus* individual subjects are used 
to generate oeasured responses* The responses are then aggregated* in 
different cocibinationa according to the research design* to produce the 

critical analytic conpariaona* In qualitative research* however* neither 

i 

a *Vrogram** nor a ^'decision" are unite that coincide with any alngular 
unit of data collection* whether the data come from an Individual or an 

13 
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txiiting record. In othar vords, the eliciting of reipoiiMs from e 
given number of individuals (the unite of dete collection) will not 
result in uy guerenteed coverage of e pre-specifieble number of pro- 
grams or drdsions (the units of analysis). 

Finally, qualitative research generally involves the use of 
stultiple data collection atratagles. For Instance » interviews, obser* 
vations, and existing documents Sksy all be needed to converge on a 
specific set of facts (Yin, 1961a and 1961b). Typically, such issues 
as %ov a decision was made*' vill involve evidence from all of these 
types of sources. Each type requires a different data collection 
strategy, and the verging of evidence from all sources also must occur 
in a systematic manner. Yet, such merging is a complex process and 
cannot b2 considered coiq»arable to the tabulation of data that occurs 
in experimental and quasi-experimental situations. 

All of these differences imply that the well-documented forms of 
research design*-e.g., those in experimental and quasi-experimental 
research— may not be appropriate for doing qualitative research. The 
traditional designs are not necessarily poorer or inept; they may siiq»ly 
be irrelevant* Ubwever, thn identification of those research designs 
that might be more relevant hat not progressed very far. Most books on 
"qualitative methods'* tend only to focus on the methods of data collec- 
tion, and not on the logic of research designs (e.g., Schatzman and 
Strauss, 1973} Fiedler, 1976; and Murphy, 1980). This problem also 
exists with texts that emphasise qualitative methods in evaluation 
research, in which research design issues are again only mlnlnally 
covered (e.g., Patton, 1980). On another front, althou^ there ia a 
well-developed debate concerning the relative advantages of qualitative 
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ys. quantltatlvt acthoda U.S.i Htrton, CoXtaani and Koaal, 1979; Cook 
and KaleUrdt, 1979; «nd S«lth ted Louis ^ 19d2), the dabatc haa not 
wually toucbftd upon rucarch dm^tgfi. isauM* 

Itt AUMtty^ nbtn InvutigAtlcmt Involve aoBe type of quidltetlve 
rftaearch» our knoirlftdse of rueerch dulifM eppMre to be Halted* 
Dbfortunetely» fhle acene tltet e lerge gap Miete» because nuaerous 
toplce In both eoclel edence end public policy reeeerch require quell- 
tetlve epproechee» Including! decleioiuieklng* progreai Impleaentetlon* 
progrem eveluetl<m» tbe tnnovetlOD proceee» Individual end orgenlMtlonel 
lif«-cyclee^ end e vide errey of leeuee In neighborhood change* economic 
developsenti Interorgenlzetlonel reletlDnshlpe» end IntergoveniDentel 
effftlre^ 

Purpose of the Present Study 

The purpose of fhe present stuify vss to Inltlete e forasl description 
end snelysls of quelltetlve reeeerch designs* This ves' done by indl- 
ceting how such designs heve be*m used In en illuetretlve toplc» 
fcnovledM utlllzetlon a The topic covers the process lAereby (Lasersfeld 
end Xeltz» 1975; end Tin end Cveltney» 198U): 

Knowledge produced by one person (or orgsnlzetlon) Is put 
Into use by enother person (or orgenlsetlon) * 

This topic provides en excellent setting for exemlnlng reeeerch design 
issues In quelitetive reseerch* for e number of ressons* 

Flrst» the topic autt be studied through quslitetlvs reeeerch 
Mthods*-ieaga» no sini^le set of expi^rlsents viU provide sufficient 
Insight for explelnlng hov the process^rlUa Second* Interest In 
knovlcdgc utiUzatloo ha* b«cn rlalng. In part dua to the continued 
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optional Itsiveitnent In basic and applied raaeaixh. This level of R&D 
effort has lad to questions regarding the ultljnate utility of R&D*based 
knovltdge, and therefore an Increase in utilization has become a 
priority policy objective. Third, the topic la not one in i^ich formal 
raaearch designs have htLu articulated. In fact» the experiences of 
ttost research Investigators have been In the opposite direction. Most 
investigators have found themselves forcei! to develop their o<wn research 
designs vithout having an ability to refer to the potentially relevant 
experiences of othar Investigators. The low degree of transfer is not 
the result of a low level of communications among such Investigators; 
rather, the Investigators have not developed a common 

vocabulary or set 

of principles for discussing research design Issues In these settings. 

With knowledge utilization research as an example, the following 
chapters of this report will therefore elucidate the prominent issues 
In research designs for qualitative methods. 

The Organization of This Report 

Most reviews of apeclfic research designs are organized according 
to the logic and potency of the designs (e.g., Campbell and Stanley^ 
1966; and Cook and Caiiq>bell, 1979). Weaker designs are differentiated 
from atronger ones, mainly on the basis of their ability to cope with 

reata to Internal validity. The resulting catalog of designs is thus 
, ^ented In a manner that allows an investigator to Identify their 
strengths and weaknesses and to choose among them. 

The preaent review of qualitative research designs was initially 
intended to mimic the existing presentations of experimental and quasi- 
experimental designs. However, no atr^g themes emerged, whereby designs 

IS 
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could bt arr«y«d «cco?^iiiB to any logical «equence. In retrospect* «uch 
w effort suet etlll be considered preuturi:« $lven the etete of the art. 
Bbtfever» tblp oheervetlon led to « sore Importut Insight: thet the 
clesslcel experlMntAl and quasl-axperliisntel designs all sssuoe s prior 
step— that the Investigator has slraady selected « resesrch strstegy» 
Which la to conduct an egperlnent * The qualitative research under review* 
in contrast* vas dominated by the use of two other research strategies 
(Yln« 1981s)~the use of aurveys (a paradox that Is explained later)* 
the use of caae studies * or the use of both. 

This Insight led to s nore spproprlste organization for discussing 
the pertinent design Issues * and such an organization dcMDinates the 
remainder of this report. Chapter II describes the problem of **Defining 
s Topic In Qualitative Research*^* a atep that Is critical and that must 
occur vhether a aurvey or case study vlll he undertaken. Chapter III 
then discusses the Issues regarding **Survey-Domlnsted Designs/* 
Chapter IV the Issues regarding *'Csse Study-£(»&lnsted Designs*** and 
Chapter V the Issues regarding **Mixed (Survey and Case Study) Designs.** 
Becsuse our review also discovered some Important principles with respect 
to project management factors and their potential effect on research 
deslgna* Chapter VI discusses these lessons under the title of **ProJect 
Kanagement and Resear;:h Design.*' The summary conclusions of our review 
sre then presented In Chapter VII. 
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U, DEFINIKG A TOPIC TM gUAUTATIVE RESEARCH 

The initial t«sk in dealing with any type of research— quantitative 
OX quail t at ive«fa«s to do with the definition of the topic to be 
atudled. In general, this involves distinguishing between a phenomenon 
and Its context, contrasting the features to be covemd with those that 
are to be Ignored (Yin, 1982b). 

The definitional task ahould not be approached from the point of 
view that there Is necessarily a singular answer in defining each topic. 
Every Investigation may define a topic in a different manner » depending 
uppn the purpose of the study. For example, common topics of inqui'ry, 
such as decisionmalcing, public programs^ or organizational change, all 
represent research topics that may be defined in a number of ways. 
There are limits to the range of possible definitions, however, and 
these are mainly imposed by the nature of prior eo^lrlcal inquiry. Thus, 
to produce a study that contributes to overall scientific knowledge, the 
definition of a topic should follow some previous theoretical base or 
earlier empirical inquiry, even if the purpose of the study is to 
challenge such theories or results* To this extent, a definition Should 
be based on some discussion of prior work on the same topic. 

The Phenomenon: What is Knowledge Utilization? 

Our own definition of knowledge utilization derived, in a pre- 
liminary nanner, from the original knowledge utilization problem. This 
problem has been stated in the following manner (Lazarsfald and Seltz» 
1975) : Htnr can research-baaed knowledge be used in applied settings? 
Such a question need not be limited tti^ research-baaed ideas, however. In 
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wmny fields » new krovlcdge can alio be created through processes other 
th«n *^Tof€Bs±onMl scientific inquiry" (Lindblom cad Cohen» 1979) » and 
auch knowledge has come to be regarded as **cra£t knowledge.** Whether 
research<^based or cra£t^based» the nore general fono of the knowledge' 
utilization problem thus leads to the Identification of three essential 
components In the utilization process (see Figure l)i 

- A knowledge producer , or a party responsible for Inventing » 
developing* or refining new Ideas; 

- A kttowledRe user > or a party responsible for applying the 
Ideas In aome setting* which nay alternatively Involve; 
(a) the making of a decision* (b) the installation of a 
new practice* or (c) Increased enlightenment over a given 
Issue; and 

* A communications process > thereby both of the above 
parties are able to establish and maintain contact with 
each other* 

As thus defined* the key characteristic of the knowledge utilization 
process is that it Involves different parties fulfilling the knowledge 
producer and knowledge user roles (Yin «nd Gwaltney* 1981a). 

Ignored by this definition are several situations that may be con- 
sidered relevant in a broader sense* but that are nevertheless not directly 
pertinent. First* knowledge utilization can be a process that occurs 
totally within the same individual. A person may Invent and develop his 
or her own idea and then put it into practice. This intra-indlvidual 
situation* however* is not the same as our problem -of getting knowledge when 
the knowledge producers and users are ^different parties. Second* 
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Figure 1 

THE KNOWLEDGE UTtLtZATtON PROBLEH 




SOURCE: Lazarsfeld and Reltz (1975). 
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knowledge utilization uy occur within the atme flna or organization. 
Which alght contain both knowledge producing and knowledge using units* 
The existence of cooBDOn corporate policies, however, nay affect the 
knowledge utilization process In a different manner than the nore 
CQonon situation, where a research investigator vbtks within on^ 
organization (typically, a university) and a knowledge uiier works 
within another organization (typically, a private business or a public 
agency) « 

Indeed, the essential characteristic of the knowledge utilization 
problem, as defined here. Is that different individuals, located in 
different organizations, serve knowledge producers and knowledge 
users« Under these conditions, knowledge utilization may be considered 
a process occurring between unlike-*--or heterop hilous (Rogers and 
Kincaid, 1961)"Unlta« Effective conmunication is likely to be 
especially difficult between these two kinds of parties, because they 
are likely to be members of different professional groups, with corre- 
spondingly different networks, norms, and problems (especially if 
research*based knowledge is involved rather than craft-based knowledge) « 
MAvertheless, as Rogers and Kincaid (1981) have noted. Innovation is 
more likely to occur in auch situations, as opposed to the situation in 
which both parties belong to the same kinds of organizations or 
professions*-i«e« . homop hilous units« 

Search for Knowledge Utilization Studies 

This basic definition was used to identify existing knowledge 
utilization studies* The search for such atudies covered a variety of 
public aervices, private industries, «^ types of knowledge* In fact, 
the aearch was unconstrained along such lines, only being limited by 
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two conditions* First* the study had to he hssed on sooie original data 
collection and empirical evidence* Senondt the atudy had to have been 
Identified as a result of aeveral search techniques* 

- Citations in existing research studies* including 
reviews of pertinent literature i 

" Identification hy various experts in the field* in- 
cluding an expert panel assembled specifically for our 
study* as well as other investigators around the 
country; 

^ Listing among awards made by various federal agencies 
having special programs on research utilization* the 
use of new technology* or exemplary practices; and 

- Appearance in relevant social science books or Journals 
over the past ten years* 

A list of all of the individuals and organizations who were contacted 
during the course of compiling this list of knowledge utilization studies 
is given in Appendix A* 

This manner of identifying knowledge utilization studies produced 
some Imprecise outcomes* At least half of the studies initially thought 
to be relevant* by title or even by brief description* C±d not In fact 
neet the basic requirements of our definition* An examination and 
classification of these non-knowledge utilization studies yielded 
Important insights into the structure of the broader literature and 
hence the context within which knowledge utilization falls« This 
definition of the context constituted ^n essential complement to our 
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MXller definition of the knowledge utilization phenonenon^ end thus 
deaervee detailed attention* 

The Context: Topics Related to Knowledge Utilization 

Our definition had cvphasized three components: a knowledge pro- 
ducer, a knowledge uaer, and a connunications process. The studies that 
did not tteet our definitional requirements were those that expressed no 
specific concern for knowledge utilization and these three components. 

For exanple, one stream of research— on the diffusion of innovations- 
was In fact ignored by our search. This literature^ quite extensive in 
nature, is primarily concerned with cosnunication patterns among knowl-*- 
edge users (Rogers and Shoemaker, 1962)- Among similar types of users — 
e«g«, farmers — some users consistently adopt new innovations earlier 
than others, and .diffusion research has attempted to identify the 
characteristics of these early adopters as well as the patterns of 
diffusion under different conditions- However* the research rarely 
attempts to draw any links to knowledge producers, nor does it give much 
attention to communication patterns between producers and users* 

A second type of non-qualifying study* also prominent in the 
literature* similarly focuses on user behavior alone, without regard to 
knowledge utilization. This is the literature on Innov&tloas or on 
organizational change . in which events to a user organization are traced, 
hut these events may not have been the result of the transmission of a 
new idea. (The studies of this sort that were Inadvertently thought 
initially to be knowledge utilization studies are listed* together with 
the other non-quallfylng atudies* li Appendix B.) For example, many 
changes occur simply because of the av^labllity of new funds or some 
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other admlniitrative or political initiative. The resulting cfaugefi 
in A local orsanizatiOQ uy be significant events in an organization 
life history. (a.g.» Tin» 1981c) » and apeclfic patterns of inplenentation 
ttay yield aigaificant Insights into organlicational procasaes (a«g.» ' 
ftemian and McLaughlin^ 1979). However* these issues are not sinilar to 
those of concern in knowledge utilization* 

Yet a third type of non-qualifying study focused only on knowledge 
production issues* vith no regard for knowledge utilization. Typically* 
the productivity and creativity of research teams operating in various 
settings has been a concern in the R&D managenent literature* The 
ultimate purpose of sucu studies is to Improve R&D management through 
such manageiDent> actions as: the employee recruitment process, changes 
in the formal K&D organization, or Improvements In published communica- 
tions among knowledge p^'oducers. However, this literature gives little 
or no attention to the ultimate application of any of the research that 
has been produced. Thus, even where dissemination projects are the 
initial forces of concern (e.g., Madey et al., 1979), the knowledge 
utilization process has been ignored if the focus of study has been the 
knowledge production process alone. Such inattention is in part Justified by 
the fact that much of the research is of a basic nature and intended 
primarily for communication to other researchers, rather than to any 
practical setting. <The studies that were found are again listed in 
Appendix B.) 

A fourth type of non-qualifying study was readily confused with 
knowledge utilization studies, but on closer examination also failed to 
meet tha definitional requirements. Such studies focused on networking 
arrangements , ^ich nay have involved '.relationships among organizations 
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(e*g*« Warren at al** 1974; «nd C«tei et al** 1981) or asong individuals 
(e*g** Saraaon et al*« 1977; ud Saraaon and Lorentt* 1979). In one 
caae« a atudy focused cm the role of professional associations as part 
of a technology^^communiciit'lons system among local aervice agencies 
(Bingham et al** 1977). Such a atudy vas not considered a knowledge 
utilisation atudy because It vas oore concerned with the health and 
viability of the networking arrangement* rather than the transmission 
of apecific Information from a knowledge producer to a knowledge user* 
Many of the networks being studied vere« of course* knowledge utilisation 
networks-^i.e** the purpose of the netirark was to exchange information 
about new ideas* However* the atudy only focused on the general 
pattern of comminlcations^-efg** what parti'^is communicated most 
frequently with which others«vithout regard to any apecific utilization 
experience. (The studies are also listed in Appendix B*) 

These four different types of studies* along with the knowledge 
utilization studies that did meet ^he definitional criteria* can be 
depicted in an expanded form of Figure 1* Thus* Figure 2 repeats the 
core elements of Figure 1* showing the communication of ideas from one 
party to another; but Figure 2 goes beyond these core elements and shows 
how the four non-qualifying types of studies are related to these core 
elements* Essentially* each of the four types represents an expansion 
of the core elements* but at the cost of Ignoring one or more of the 
other elements ud of focusing on some problem other than the knowledge 
atilization problem. Figure 2 thus* illustrates the knowledge utilization 
phenomenon embedded within Its context* as phenomenon and context have 
been defined fot the purposes of the present study. Having defined the 
distinction between phenomenon and context* ve can now proceed and 
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ftxamlae the phenomenon Bore clotely^l^e^ , exjuoilne research design 
l«iues vith regard to knowledge utilization studies « 

KnowledRe UtUlaatlon Studies Identified 

Given our aearch process and this distinction between phenomenon 
and context. 32 atudles of knowledge utilization were Identified^ These 
atudles and their characteristics are listed In Table 1 (a full set of 
citations Is given In Appendix C)* 

Of these 32 studies, the following were the salient descriptive 
characteristics > 

^ About 2/3 were published In 1980 or later or were still 
In progress (see column 1, Table 1); 

About 1/2 covered the field of education, with the other 
half being distributed among a variety of service or 
policy topics (see column 2); and 
- More than X/2 relied on a case study resesrch strategy, 
with the remainder using a survey strategy, for 
collecting the pertinent evidence (see column 9^* 

No doubt, the recency of the majority of the studies reflected the 
Increased interest In knowledge utilization In the past few years « 
Although many studies had been conducted. In earlier periods, regarding 
the four nonqualifying topics, knowledge utilization can be considered 
a topic that has only lately received solid empirical attentlon**in 
aplte of insightful observations made years ago (e.g., U«S. House of 
Representatives! 1967) < 
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The donlutlon of atudiea on eduMtioul topics^ vm8 al6o not sur- 
priiing. Both of the predecessor conponents to the U^S* Department of 
Education^i^e. » the Office of Education and the National Xnttitute of 
Education~have Ibaen the moat prominent agencies that have supported 
knowledge utilization research* This Inference extends to the other 
higher frequency toplcs^urban aervices and health (and mental health) — 
reflecting the activities of research-funding agencies such as the 
National Science Foundation and the National Institutes of Health (and 
the National Institute of Mental Health) » respectively* 

What was modestly surprising in this final array was the distri- 
bution among research strategies* £ven though knowledge utilization may 
be^'considered a topic In qualitative research* many studies relied 
solely upon the use of surveys » and many others used surveys in combi- 
nation with case studies* Even though the case study strategy was 
d(»&inant» this high frequency of surveys deserves explicit attention* 
Thus» the following chapters are devoted to discussions of research 
designs under three conditions: when surveys dominate (Chapter 1II)» 
when case atudies d(»&inate (Chapter IV) » and when both are used 
(Chapter V)* 

From a research design standpoint* this chapter has illustrated 
the key atep of defining the topic of inquiry* The definition began with 
« link to previous research and theoty» indicating the significance of 
the topic €0 be investigated— i*e* » the knowledge utilization problem* 
The definition was then applied, through an operationally specifiable 
aearch process, to Identify various nofllnations for studies potentially 
falling within the topic of inquiry. 
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Further examination then shoved how iome studies, initially 
Identified, actually fell outside of the topic. These studies focused 
on aone related toplc-*c.g.» organizational change, R&D nanagenent, or 
metwrfcing~but did not consider the essential knowledge utilization 
relationship betMen producers and users. In this sense, these studies 
formed one context for the knowledge utilization problem. Illustrating 
how context and phenonienon need to be distinguished, even If In a some- 
^at arbitrary manner. 

Finally, this entire procedure yielded a set of 32 target studies, 
all of t^lch fit the definitional criteria. The chapter thus Illustrates 
the process of operationally defining a topic of Inquiry, and how 
specific examples of items falling outside of and within the topic can 
be used to bolster the definitional process. Such an approach, whether 
applied to an analysis of previous studies (secondary analysis) or to 
an original entplrlca? inquiry (primary analysis). Is a necessary pre* 
liminary step in designing qualitative research. The sub;;equent steps 
are described In the following chapters. 
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III. SDRVEY'PQMIHATED DESIGNS 

The use' c£ ft survey to collect evidence ebout knowledge utilization 
h»B been ft commcm technique. Tftble 1 previously Indicftted that half of 
the 32 knowledge utilization atudies under review had used surveys* with 
y based on survey data alone and another 9 combining surveys with case 
atudies* 

On the surface* the use of surveys to investigate a topic such as 
knowledge utilization would seeai anomalous. This is because of the 
previous characterization of the topic as having no clear boundaries* 
no easily specifiable unit of analysis* no relationship to a singular 
unit of data collection* and a need for multiple data collection 
strategies (see Chapter I)* Nevertheless* the use of surveys was found* 
and survey evidence did contribute to an understanding of the knowledge 
utilization process. Bow such a contribution was managed* and the 
limitations of the approach* are described In the present chapter. 

An initial task* however* is to define lAat is meant by a survey. 
Three conditions Appear to be relevant and were used in our review: 

1. The source of information is an individual respondent * 
who in turn represents the major unit of data collec- 
tion and data analysis; 

2. Each such respondent is selected on the basis of some 
explicit samplina technique* logic* or is part of a 
survey of the entire universe* *nd 
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3* the data collectioo i8 based oo the use of e 

•tructured instrument* in vhich each respondeat Is 

soked the same questions (vhich nsy» however^ be 

open- or closed-ended). ' 

Where these three conditions vere not ttet» the distinctive characteristics 
of a aurvey vere obscured beyond acceptable limit'?* Thus» the three conditions 
appear essential^ at least in the present reviev. Beyond these three 
conditions^ survey technology may (and did) vary considerably* 

The following discussion focuses on those situations in vhich 
surveys dominated a knovledge utilization study* In other vords» sur- 
veys vere used as the main source of evidence^ rather than in conjunc- 
tion vith case studies. Typically* the surveys investigated either 
knovledge producers or knovledge users* but not both* Furthermore* some 
surveys also made a prior specification of the knovledge utilisation 
event or medium^^ vhile others did not* (Such specification meant that 
a respondent vas a&ked about a specific report* idea* technology* or 
interaction that had been identified prior to the onset of the study*) 
These three approaches represent major types of survey designs* and are 
described below* 

Surveys of Knovledge Producers 

As previously noted (see Chapter II)* studies that only focused on 
knowledge producers vere not included in our definition of eligible 
studies to be revieved* The ineligible studies vere those that vere 

concerned only with various facets of the knowledge production process*- 

r 

i*e** R&D inanagement«but not with knowledge utilization* 

315 
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A f«w eligible Atudifts, in contrast, llAlted tfa«iittelvefi to data 
collected fron knotfledge producers, but aimed at a broader aet of 
questions regarding the entire knowledge utilization proceas. This was 
wpacially found in two of the aurveys under review. Xn the first, 
Myara and Marquis (1969) interviewed respondents in 121 firms. £ach 
respondent was asked to Identify the sost significant technological 
innovation that had been produced by the fins during the most recent 
5*10 year period. The interview then continued into the topics of how 
the innovations had been developed as well as the marketing strategies 
that appeared to have accounted for success. 

The aecond study involved a survey of research investigators in the 
U.S. Forest Service (Llngwood, 1979). The personnel were given a self- 
administered questionnaire and were asked to describe the most effective 
dissemination atrategies for their research. The most frequent response 
was that the research should be based on some sensing of client needs» 
and that the research should be conducted in a timely manner. 

In both atudle&» no other data collection effort was made. Thus, 
the inferences about the knowledge utilization process were based solely 
on the perceptions of one group of persons»the knowledge producers. No 
attempt was made to ^corroborate these perceptions in comparison to 
other perceptions (e.g., of the knowledge users in the same situation). 
Similarly, no atten^t was made to corroborate the perceptions with the 
question of how utilization actually occurred, as might be possible by 
uaing other types of evidence. Such a characteristic of producer*^ly 
Surveys must be considered a serious weakness. In fact, without 
corroboratory evidence, one might even auspect that the producers^ 
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percaptioM «tre not Accurate reflections of the actual utilization 
procea«« and that any cqdcIusIoos about the proceas alght be faulty* 

Surveys of Rnotfled&e Uaers 

Surveys were also conducted by obtaining information from knowledge 
uaera. For exanple* in a aajor study of the use of social science by 
federal policy officials* C^plan et al. (1975) queried 204 high-level 
officials in federal agencies » aekiog them for specific instances in 
i^ich they had used socisl science knowledge* For each instance of 
use* the officials vere also asked for corroborating evidence* and the 
quality of this evidence was also taken into account in estimating the 
extent of use* The results of the study shoved a surprisingly high 
degree of utilisation (74 percent of the respondents had at least 2 to 4 
instances of use with good corroborating evidence). 

This type of design also was used in a survey of social service 
agencies in England (Rothman* 1980)* in ^Ich knowledge- users vere asked 
about the circumstances in vhich the findings from some recent report 
had been put to use. Xn this situation^ utilization involved an appli- 
cation to service practice* rather than to policy decisionmaking (as in 
the case of the Caplan et al. study)* but the basic research design vas 
the aame. Similarly » as another example of an application to service 
practice » but In the business sector* Desfapande (1981) surveyed 
executives regarding their use of the most recent market research study 
coomisaioned by their firm. 

In these and other user surveys^ three features vere prominent* 
First* the user aurvey vas a good td^ol for estimating the gi^tgtit of use 
of aome aort of knowledge* Second^ ho^er* the user survey had the 
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•atte shortcomings as the producer survey in yielding inferences about 
the knowledge utiliMtion process^-i^e^^ the survey only brought one 
set of perceptions to beer on the topic* Third* an additional short- 
coming stemmed from the feci that no attentat was made to assure that 
the knowledge event or medium* as reported by the knowledge users* was 
the same in all cases* Thus* even such traditional survey items as 
"user satisfaction*' are difficult to interpret* because the collection 
of knowledge events or media may vary in some unknown fashion* In 
contrast* where the investigators initially identify a set of specific 
events or media^ more inferences can be made about user satisfaction 
and the potential implications for the utilisation process* 

Surveys with Prior Specification of Knowledge j£vents or Media 

This last shortcoming can be overcome by naming specific reports* 
ideas^ technologies* or interactions at the outset of the investigation — 
i*e** knowledge "events'* or media—and then conducting a user survey* 
Such a design was followed in several of the studies under review* 

For example* Weiss and Bucuvalas (1980) surveyed 155 decisioniDakers 
and asked them about each of 50 pre-specified research reports* The 
decisionmakers* responses covered their degree of use of these reports* 
their reasons for such use (or non-use)* and their other opinions about 
the reports* Because 50 specific reports bad been identified and rated 
by all of the 155 respondents* Weiss and Bucuvalas were able to compare 
the results for the same types of reports* and thereby to develop 
stronger Inferences about user satisfaction* 

In principle* the naming of a specific array of reports would also 
' have allowed the investigators to compare the user responses to two 
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other AourcM of lti£onitlOQ**fhe observed characteristics of the 
rcporta tbeuelvesf as well «s reaptmses by thoae vho had produced the 
reports (knowledge producers)^ This vore potent design was not used« 
faovevcr^ Nor was It used In another study in which the data cwue from 
Scnowledge users* but where a prior apeclficatlon of knowledge events or 
aedla slIso had been made« This atudy was by Levlton (1961), who sur- 
veyed the users of a apeclfic aaiDple of evaluation studies In health 
care finance and in education; these users had been Identified either 
by the project officers or the research Investigators of the studies* 
but no attempt was made to collect systematic data from these other two 
source s« 

One can only guess at why the more potent design (survey-of-u$er$« 
combined with survey--of*producers) has not been more frequently found 
among the survey-domlnated studies* To conduct auch a study* the most 
li&portant Item is the pre*namlng of a specific knowledge event or medium 
(report. Idea* technology* Intera'itlon* etc*)* Such a named event or 
medium then allows the Investigator to be sure that the users and pro- 
ducerit have the same referent when producing their responses* However* 
one possibility is that this design has not been used more often because 
of Its potential costliness; in general, as shall be shown In Chapter V, 
this design was found among those studies employing both surveys and 
case studies* and thus those studies funded at substantially higher 
levels of effort than the studies covered in this chapter* 

Simmary: Surveys as a Research Strategy in Qualitative Research 

These experiences now permit some tentative conclusions about the 
use of surveys aa the aole research atvategy for investigating a 
qualitative research topic auch as knowledge utilization* 
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With sdequate definition* surveys sre helpful la determining the 
prevaletice of « phenooenon mong « known pool of respondents. Thus* 
the extent of use is s portion of the knowledge utilizstion process 
thst can be estsbllshed through the survey aethod* even though the 
topic renslns sn essentislly quslitstive one. Other aspects of the 
utilisation process oay sl«o be Investigated* but only under sdditional 
constraints. In particular* If s survey begins vith a pre*na]&ed event 
or ttedium* perceptions of users or perceptions of producers can be 
aggregated and interpreted. If the suae referent Is used vith both 
groups* the perceptions of both groups nay even be compared vith each 
other . 

In general* the survey strategy is therefore appropriate for dealing 
vith either of tvo aspects of a qualitative research topic: issues of 
frequency * and Issues of perception and attitude . In both cases* how- 
ever* the use of the survey sssumes thst Individual persons are the most 
accurate source of InforiDatlon* and one can Imagine contrary situations 
vhere survey probes might not be sufficient. For Instance* the extent 
of use of a given technology or idea might be reflected ±n some mechanical 
rather than huinan system — as In the amount of computer time spent on a 
particular program. In '^uch s case* the archival record of the computer 
time would be the more accurate source of Information about the extent 
of use* but the searching and analysing of these records would require 
some resesrch strategy other than a survey. Similarly* the utilisation 
of knowledge In education may Involve students rather than teachers as 
the ultimate **users*" yet the most common survey design usually relies 
on teachera* reports of student behavior* and such reports might again 
not be the most sccurste source of in/braatlon. 
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Beyond the Usucs reUted to frequency, perception. «nd attitude, 
the turvey strategy appears to have United application In dealing with 
« qualitative reaearch topic. The aurvey i« not the most desirable 
tool for inveatigating a behavioral proceafc auch as knowledge utiliza- 
tion because the asaessment of actual behavior cannot be based on eurvey 
evidence alone. In this sewe. the survey will rarely yield Information 
that can lead to the design of effective iJiterventions in auch behavioral 
processes. This type of "how" -and "why" inquiry, in contrast to the 
sheer "f riqucncy" Inquiry, depends on the use of other research 
strategies, to which we now turn. 

Re^ rence ff *■» Chapter III 

/II citations in this chapter may be found in Appendix C of this 
report (see pp. 86-88). 
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IV, CASE STODY-DOMmATED DESIGNS 

The ttst^ of cue •ttidiaa appeara to be the aost frequent atretegy for 
studying knowledge utillsetion* Of the 32 Inveetigations reviewed in' the 
preient report « 16 used CMe studies elone end soother 9 used esse etudies 
In ctxibinstion with surveys* Thue, about tvo*tbir^^s of these previous 
investigsticms used esse studies to sxsmine some aspect of knowledge 
utilisstion* 

In contrsst to surveySj esse etudies any bt considered investigations 
where the following conditions prevsil: 

. 1* No clesr boundsrv exists between the pbenoinenon being 
studied and its coDtext**s situation thst typicslly 
attends the scope of inquiry end thst produces Bore 
vsrishles thsn dsts polnts~nsking atstisticsl snslysis 
virtuslly irrcZevsnt (Yin. 1981b ; and 1982b)* 

2* There sre aultiplft sources of information . Including 
dsts from Individual respondents* on*site observstions» 
and snalysia of vritten documents and other artifacts* 

3* To deal In part with the preceding two conditions* 

dsts collection is bssed on s protocol* hot;ever formsl 
or informsl» which guides the collection of evidence* 
In a sense» the field Investigstor is the "Instrument/' 
being responsible for trsnslatlng field iMomstion Into 
the response cstegories rsquired by the protocol (Ssndsy» 
1979; and fii tiott snd Firestone, 1982)* 
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Th« frequent of case atudies in knowledge utilization reeeerch stems 
mainly from the «atcb between these ccnditions and the knowledge utiliza- 
tion process^ Because the ut^.liration process is a coo^Xex organizational 
and interpersonal procesSf the phenoinenon is not easily separable froiti its 
contexts Thus» the process is difficult to trace in a pre-designed 
fashlOHf and an investigator uy need discretion to pursue new leads as 
fieldwork progresses^ Such leads may require further Interviews* obser* 
vationsf or documentary analysis^ To conduct this data collection 
properly* the Investigator should nevertheless follow a systematic search* 
and-corroboration process* and a veil-designed protocol is one way of 
striking a balance between the necessary discretion and desired 
systematization^ 

Anong case studies* a common design issue is t^ether to cover a 
single or multiple case (Yin* 1961b) « In theory* a single case* like a 
critical experiment* can offer sufficient insights and evidence to test 
a major proposition^ Bomver* because case studies cannot be manipulated 
like experiments* multiple ^s>ies need to be used* in part* to provide 
multiple replications of the same phenomenon* end in this manner overcome 
some (but not all) of this limitation^ For this reason* the simplest 
type of multiple-case design is a direct replication design (Uersen and 
3arlov* 1976). 

Mot surprisingly* all of the case studies of knowledge utilization 
were of the multiple-case variety, iiow the case studies were designed* 
including some comment about pitfalls that may be avoidable In the future* 
is the topic of the present chapter^. TWo 5«n"al typea of designs* both 
of vhich were applied In multiple-case. situations » were found: >holistic 
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dMigus and ««bedded detigns* These types are discussed belav* followed 
by s further discussion of sultiple-csse designs more generally* 

j jrfiolistic Designs 



Q^olistic designs are those In Which the esse stu^ establishes only 
Alniul boundaries between the phenomenon-^i^e* , utilisation—and its 
context* The case atudy is typically about a large organization, such 
as a school district (e*g*, Alkln et al*, 1979) » or about an Inter- 
organizational arrangement, In which sone organizations produce new 
knowledge and others use it (e^g** Vamecke^ no date; Hoore et al*, 1977; 
Parker, in progress; and Yin and Heinsohn, 1980)* In one other investi- 
gation, the case study was about a type of innovative program or package 
(Campeau et al*, 1979)* 

The main characteri^itic of iiie 'holistic design is that It does not 
pre-specify a particular type of utilisation incident or experience* For 
Instance, the investigation might focus on a general type of knowledge— 
Sfgft the use of tests and evaluation by achool practitionerS"but make 
no attempt to define a specific test or evaluation to serve as the 
referent for the liivestigation* On the contrary* the investigation would 
mainly deal with the general relationships among knowledge producers and 
users* This type of study was done by Alkin et al* (1979)» who studied 
five school districts In which evaluations of Title I or Title IV-C 
pror rams had been produced* The resulting five case studies were reported 
In separate narratives* and the authors used these five cases to derive 
general conclusions about the utilisation process* 

Vhen holistic designs are used* ^major problem that must be 
addressed Is the potential bias In the data collection activity* Because 
of the vholistic design, a fieldworker has far-ranging discretion in 
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deciding what events are r«l«v«nt and irrelevant to the case study. 
Pnder these conditioDB^ the fieldvorker that does not follow a consistent 
path of aaarch-and*corroboration nay produce an idiosyncratic and biased 
account of events. An Important way of dealing vith auch potential ' 
biases Is to have the draft case atudy reviewed by the aajor field 
informanta. Normally^ these Informants are able to call the investi- 
gator^a attention to aome matters that may have been overlooked In the 
initial data collection. This type of corroboration with the field 
informants is therefore much more than a matter of professional courtesy ; 
the procedure plays a critical role as a validation procedure (Schatzman 
and Strauss» 1973» pp. 133*134). 

In one of the knowledge utilization studies* great care was taken by 
the Investigators to implement this procedure (Alkin et al.f 1979). A 
distinctive characteristic of this study* in fact* is that the final 
document contains the actual critique provided by the field Informants* 
as veil as a summary of the steps taken by the investigators to deal with 
these comments. In another investigation (Yin and Helnsohn* 1980)* the 
draft case studies were also reviewed several times by the main Informants* 
who identified missing evidence and Incorrect interpretations. Few of the 
other atudies under review Indicated how this procedure of using field 
Informants as reviewers was Implemented* if at all. 

The holistic designs also can lead to another problem that is i»t 
as easily addressed. Because the holistic design does not specify* at 
the outset of an Inveatigation* a limited domain of inquiry* the entire 
nature of the study may in fact shift. This seems to have occurred in 
et least one investigation (Hoore et aJ..* 1977)* which covered six case 
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studies of technic*! MSistance netvorks* Eecfa network consisted of ^ 
interorgenlzetloDsl errangeoeott in ^ich one of the organizations had 
the role of providing assistance to the other organizations. Although 
the initial rationale of the study was to Investigate the knowledge 
utilization process* and although the investigators did trace two ongoing 
assistance projects in each network* the final docui&ent tended to focus 
Qore on the activities of the assi8tance*providlng organization* and less 
on the knowledge utilization process. The final conclusions were doml* 
nated by concerns over the institutional aurvival of the assistance- 
providing organization* including such issues as the auspices for the 
organization's founding* the nature of the organization's leadership* and 
the identification of funds for support. Conclusions about the knowledge 
utilization process were then presented* but only as a lower priority. 
Overall* each case study had shifted* focusing not on the interorganiza*^ 
tional arrangement but on one of the components in the arrangeiDent-*i*e.* 
the assistance*providing organization. 

Such slippage in the general scope and purpose of inquiry can be 
addressed* even when wholistic designs are used* in at least two ways. 
First* the entire case study can be based on an explicit and firmly based 
theoretical framework"i.e.* one positing Important causal connections 
and not merely a description of events (Kaufman* 1958). The Yin and 
Heinsohn (1980) study* for instance* was about the usefulness of findings 
from major research projects in /«e field of aging. The case studies 
traced the knowlege production process (which occurred In a university 
setting) and the knowledge utilization process (which occurred in policy-* 
making or service settings). Each caa* study began with aeveral 
alternative "oodels" of knowledge utilization (Weiss* 1979)* each 
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predicting a dlffertnt aet of events* The ensuing case atudy, chough 
vholistict had as its aaln task the establiahnent or denial of these 
ecvanta* In ^hia way, the aodela provided a atrong theoretical fraoevork, 
ao that the Inquiry remained faithful to its original design* even Where 
unexpected events were discovered* 

Second* a quasi^experlDental design can be used as a partial substi- 
tute for a theoretical framework* The Wamecke Gio date) study* for 
Instance* covered 7 case studies of health care networks* in Which groups 
of hospitals collaborated to Increase the use of certain techniques of 
cancer-patient care* Each case study followed a pre-^post design, in 
which data were collected before and after a specific intervention'-- i*e* * 
a formal* collaborative progras-^-4ad been installed* The nature of the 
program therefore provided strong guidance regarding the relevant events* 
and again the inquiry was able to remain faithful to its original design* 

In summary* the Wholistic design involves the identification of 
some "site" (organization* interorganizational arrangement* etc*) at which 
knowledge production and utilization have occurred* A case study strategy 
Is used to examine the events st such sites* but there may be little 
prior specification of the relevant or most salient events* Under these 
conditions* the Investigator has a vide*range of discretion* during the 
data collection phase* regarding the events to be pursued and recorded* 

To minimize the potential biases in such an inquiry* two procedures 

may be used* First* the draft case studies can be reviewed by the major 

informants* to sssure that obvious facts hsve not been overlooked* 

Second* the esse study can employ iu strong theoretical or quasi-* 

experimental framework* which will prtc^lde additional guidance for the 

t 

dsts collection sctivlty* If these two procedures sre used* a Wholistic 
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dttiga can be « potent explanatory tool» producing criticel Insights into 
bow and knowledge utilisation occura^ 

Embedded Deeigig 

An embedded design is one in vfaich a amaller unit of analysia is 
contained (or ea^edded) within n larger unit of analysia^ For cAse 
atudies of Imowledge utilication, the amaller unit of analysis might be 
a pre-6pecified knowledge utilization event~e«g«» an interpersonal 
transaction or a particular knowledge utilization service^ The larger 
unit might still be an organization or organizational Arrangement » akin 
to those found in wholistic designs^ 

Several exaioples were founds among the investigations under review^ 
of case studies that used embedded designs^ In education* three related 
investigations all covered different kinds of Interorganizational 
arrangements (Chin» 1981; Uavelock et al«, 1981; and Tin and Gwaltney» 
1981b)« Each arrangement therefore served as the large'r unit of analysis^ 
Within aach case study, however^ specific types of events or services 
were identified as amaller units of analysis^ For the Chin (1981) study, 
these were called "transactions^" and data were collected regarding the 
role of the knowledge producers and users in auch transactions^ In the 
Uavelock et al« (1981) study, a similar procedure was followed^ but the 
conceptual label vas an *'epiaode;" and for the Yin and Gwaltney study 
(1981b)» the label was a "service/' Outside of education, an embedded 
design of virtually the aame nature vas used to study nir and water 
quality aervices» and knowledge transfer between university and local 
government (Eckfield et al.» 1978). As a final variant^ Pntton et al. 
(1975) did case atudies in the field of health. In which n specific 
evaluation study vas identified as the amaller unit of analysis* 
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The embedded design serves u an Isportut device for focusing a 
esse study inquiry^ even vhere s theoretical or quasi-sxperlnental 
fraoework uiy be absent. FurthenDore* the smsller units of snalysie can 
often involve nunerous discrete events* creating the potential for 
quantitative analysis within each case study. For instance* one may 
tabulate the number of transactions or episodes* develop explicit 
measures of their characteristics* and use sultivsriste techniques to 
analyze the data. Such an approach was In fsct used In another investi- 
gation with en embedded design* vhich examined local aervices at 18 
aites (the larger unit of analysis) and over 2000 "episodes" of knowledge 
utilization (the smaller un^t of analysis) at all of these sites (Pelz* 
1981). 

However: erne pitfall In using the embedded design is that the smaller 
unit of snalysis nay be the only level at vhich analysis is conducted. 
This means that the final case studies do not return to the larger unit 
of analysis* snd that no evidence is actually synthesized st the case 
study level. This situation represents an inversion of the embedded 
design* and potentially destroys the entire case study design. Thus* 
the Pelz (1981) investigation made no attempt to synthesize information 
sbout the original 18 sites (for esch site Indivlduslly) but effectively 
became a study of utilization episodes. SimlXsrly* another investigation 
covered 18 achool districts thst had been known to have used tests and 
evaluations successfully (Kennedy et si.* X980). The investigation's 
ujor import* however* did not synthesize the infomatlon for each 
district into case atudies* and therefore did not try to explain the 
utilization pro»ss as it occurred In >eny of these districts. Instead* 
the investigators ulnly relied upon their interview data (end ignored 
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other Aources of evidence)* end aggregated interviews of like-^iidivlduals 
(a.g. ^ achool principals) acroaa aites. The resulting presentation nerely 
provides infornstion ehout how euch like-individuals felt ehout the 
utillsetion process* in e ttanner QK>re einllar to survey than case study 
analysis. Such e transition* etertlng with 16 cases hut ending with 
evidence aggregated according to individual respondents* Is a mixed type 
of design that nay not take full advantage of either the case study or 
the eurvey atrategy* 

Multiple-Case Designs 

The distinction between wholistic and embedded designs pertains to 
the conduct of individual case studies. In other words* a design may be 
followed within each case study that is either wholistic or embedded. In 
all of the knowledge utilization investigations* however* multiple-case 
atudies were conducted. Such situations raise another important facet of 
case study design — i.e.* the problem of selecting an array of cases to 
be included in a multiple-case study. 

Maximiging Statistical Generalizability . Among the studies reviewed* 
the most common rationale emphasized statistical generalizability* 
akin to the concern for external validity in experimental and quasi- 
experimental designs. Cases were aelected ao that they somehow repre- 
aented different kinds of situations--e.g. * organizations of different 
alzes* locations In different regions of the country* services to different 
types of population groups^ and the like. Such a rationale was not sur- 
prising » because it is frequently U9ed whenever multiple-^case atudies are 
conducted* vhether on knowlege utilization or on acme other topic. 

The concern for case atudy generalizabllity has mainly been a 
reaction to the conmon atereotype of the single-case atudy* in which 
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rt«ultt tuive bttn challenged beceuie of their Idloeyncretic nature 
(Kennedy* 1979)« Even vhere robust WMures and research procedures 
have been followed » the aingle-caae llsiltations appear ao aevere that 
the acientific or even policy value of the results has been questioned* 
Thus p miltiple*caae atudies have been used itore frequently in recent 
years as one reaponse to this problem (Herriott and Firestone* 1982)* 
Depending upon the available resources » investigators may aet out to 
conduct as nany case studies as possible* in the hope of nkaximlzing the 
generaliz^Mlity of the results^ 

This rationale is nevertheless weak and not necessarily preferred 
for «ultiple*case design^ Mo matter how many cases are included in a 
study » the number of cases vill not be aufficient to reduce the 
generalizability problem to any aubstantial degree* This is because 
each case study represents a complex* multivariate situation* Any 
sampling plan that includes even auch straightforward a variable as **city 
aize/' for instance* will be distorted by differences in jurisdictional 
structure (some "cities" include an entire metropolitan area* while most 
cities only Include the central city)* degree of overlap between the 
organization(s) being studied and the city area (some achool districts 
a<rve a vhole city; others serve only part of it)* and other similar 
complications « In like manner* every fscet of a case study vill produce 
a large number of exceptional circumstances in addition to a vety large 
number of potentially relevant variables* Under these conditions* no 
reasonable number of caae studies vHl enable u investigator to deal 
effectively with the generalisability problem (Kennedy* 1979). 

Maximizing Analytic Generalizabi^ty * An alternative rationale was used 
in one atudy under review (Szanton* 1981) and represents a potentially 
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Importan: precedent* This sultiple*cese design enphesizee keneralizebiUty, 
' but to e eet of theoretical propositions retfaer then to e population group* 
Such analytic generalizability, in contrast to atatistical teneralizability, 
la applicable to virtually every case atudy topic ^ and not Just knowledge 
utilization (YiD, 1981b; and 1982a)* 

The ttultlple*caae logic is similar to ttultiple-experiment logic 
(Bersen and Barlov, 1976)* First, aeveral cases need to be conducted that 
are direct replications of each other, to establish that a set of results 
is robust* Second, additional groups of cases can be added, as critical 
tests of various alternative theoretical positions* For example, the 
Szanton (1981) study begins with eight case studies, all shoving how 
university groups, in different circumstances, have not been helpful in 
providing advice (knowledge) to cities* This initial group of cases is 
repetitive enough to convince the reader of the soundness of the general 
proposition* The study then attempts to '^explain** the reasons for such a 
dismal record, suggesting next that the academic setting Is not the 
causal agent, as might be expected initially* This proposition is examined 
through the presentation of a second group of five case studies, all 
Involving independent research organisations that have also had difficulty 
in providing advice (knowledge) to cities* Yet a third group of cases, 
not enumerated in detail, correspondingly Indicates, however, that 
university groups have been helpful to other aectors— »to businesses, 
engineering firms » and even federal ^d state governments* Finally, a 
fourth group of three case atudies, focusing on "change agent** organiza- 
tions. Indicates that advice can be jsuccessfully given to cities ^en the 
knowledge-producing organization acts 4f a change agent and Is not merely 
acting as an advlaer* The atudy thus concludes that the advice-giving 
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pToctM must Mtch the nceda of the city gover!u»^at&» Which generally 
require iDtereets in Impleaeatetlon and not aerely advlce^glvlag. Given 
fhls c3tplenatlont ualverslty fallurea to assist cities cannot be 
attributed to the Inherent characteristics of academe. Rather* universi- 
ties (and aoaunlverslty groups) will fall ^enever a change agent role Is 
ahunned* 

In all» 16 caae atudles divided Into three groups are enumerated la 
the Szanton atudy* vith an unspeclflable number represented in a fourth 
group* This ttultlple-case design Implicitly attempts to provide direct 
replication within group» and analytic generallxabllltv across groups* 
What binds the individual case studies together* in logical fashion* is 
an evolving theoretical argument* much like the logic underlying multiple 
experiments. Though the Szanton study 1$ not necessarily atrong in the 
other facets of its research procedures* this single example of a 
multiple-case^ replication design is an exemplar of how multiple-case 
atudles can be most effectively designed in other knowledge utilization 
atudles (Yin* 1962a). 

Summary 

The case study was the most common research strategy used in the 
knowledge utilization investigations under review* This chapter has 
discussed the two most prevalent types of case study design* both 
applicable to the conduct of individual case studies: Whollstic designs 
and embedded designs. Each type has some advantages and disadvantages* 
but each is able to deal with the entire knowledge utilization process 
in an explanatory manner. Because all of the investigations under review 
were multiple-case atudles (and not aii^ly alngle^case studies)* two 
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logical ApproACbea to Bultiple^jisc design have Also been discussed* the 
generalissbility and the replication designs. The latter was found to 
be a potentially nore fruitful rationale for future atudies than the 
forser design* 

A major ahortcomlng throughout th.i use of all the^e designs* however* 
is that case litudies alone are not good tools for establishing the 
frequency or extensivencss of a particular phenomenon* Although great 
Insight was derived into the knowledge utilization process* none of the 
case studies attempted to assess the degree of utilization that had 
occurred* This gap is filled veil* as Chapter III suggested earlier* by 
the survey strategy. For this reason* one of the more intriguing types 
of knowledge utilization studies Is that which combines surveys with case 
studies* and these (nixed) designs are covered in the following chapter. 
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V, MIXED (SURVEY AND CASE STUPTf) DESIGNS 



As nor:ed throughout the previous chapters, nine of the 32 studies 
used Bixed designs, in vhich both surveys and esse studies existed as 
resesrch strstegies* These nlL: studies should be sugmented by a tenth 
study (DlHaggio and Useem, 1979), vhich we originally classified as so 
investigstion that relied aolely on case studies (see Table 1), but vhich 
in fact was s follow-up to an earlier study that was based on a survey 
atrategy* In this aense, the study was in fact part of « mixed design* 
and ten studies therefore serve ss the basis for the present chapter* 

* Mixed designs are potentially the most conrprehensive designs* 
because they enable an investigation to address different sets of questions* 
vhich Chapters III and IV have shown to be coioplementary* The survey 
strategy can provide information about the frequency of a phenomenon and 
about the perception and attitudes of key individuals (Chapter III); the 
case studies can provide insight into the "how and vfay" of a ':omplex pro* 
cess (chapter IV) * Moreover, aixed designs may be considered more 
sophisticated than either the aurvey-dominated or case study-dominated 
designs, if only because the mixed designs sttenpt to synthesize the 
survey and case study strategies* 

;(*svertheless » the management and implementation of mixed designs 
can be difficult* level of resources is required that goes beyond the 
almple survey or case atudy atrategies» and these resources must be 
integrated in a productive and timely manner* Possibly for this reason, . 
investigations vith mixed designs can be costly and can take seversl 
yeara to complete* And not aurpriaingly» the ten studies under review 
Include the most expensive of sll 32 investigstions under review, vith 

56 



50 



Mveral having bftas funded At m level of $600,^000 or florae » and with one 
etudy having involved $2^8 million^ 

Four types of alxed designs were foimd and are diitcusaed below. 
1!heae are: parallel designs* aequential designs* ca&e&^vithin^surveys 
deaigns* «nd aurvays«vithin->cases designs. 

Parallel Designs 

A patallel design is one in lAich the case study and survey com* 
ponents of an investigation are not really In&egrated In any technical 
aense. Although each efforc is alised at a net of conceptual objectivet 
that nay be complementary* each effort is undertaken rather Independently, 
in terms of Internal design and instrumentation. In other words* the 
units of analysis* measures* and analytic comparisons are different and 
cannot be aggregated or compared in any analytically formal manner* such 
as the techniques described by Glass* McGaw* and Smith (1981). However* 
the parallel design may involve varying degrees of man^cgerlal integration* 
in which t-zrzzt^gBtOTG Involved In one component develop their plans* and 
later their conclusions* in consultatioo with each other. In this latter 
regard* the investigations under review varied. 

For exas^le* ooe studty of the uti3j.2ation of urban technologies in 
fact consisted of nine substudies* according to the original design pXan 
(Doctors et al.* 1979). Each of these nine aubstudies appeared as a 
aeparate Investigation* with its own research objectives* dtsc^ign* and 
data collection plans. In particular* one aubstudy* relying wre heavily 
on aurvey methods* compared the benefits derived at 27 sitea (as a result 
of the availaMllty of technology essistance) with a "control** group of 
27 non^aaslated altes. This aubstudy vas to result mainly in a cost- 
benefit analyaia* Indicating the advantages* If any* of using the urban 
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ttchnologlM. A titcoQd ftub«tvdy» relying core heavily on c«9ift #tudy 
mthods* focused on 15 iimovetlotis « to detemice Uow end why clijtnge 
occurred with these Innovations i including the Investigation ot knowl- 
•dg^* production and knowledge utiUration factors. Presumably* each of 
these case atudies could include events occurr 'tsg at aore than £ single 
site» hectu!»e the saine Inncvatlon night have heeu used at more than one 
site» and In this sense the survey and case study designs involved 
different units of analysis. The degree to ^ich the investigators of 
these different substudies consulted «ach other is not knovn» so the 
overall study nay have remained a collection of substudies. 

As a second example » a large*5cale study of knowledge utiJ,ization 
in education alsc Involved a number of s^arately identified substudies^ 

* A study of federal and Atate-level programs and knowl^ 
edge production activities; 

* A study of one specific program (the National Diffusion 
Network) » designed to produce and utilize new knowledge 
created by practitidnera; 

* A study of the behavior and attitudes of external 
change agents, or linkers; 

* An analytic effort designed to ser/e in a formative 
evaluation mode* to assist school districts; 

- A study of 145 school districts and the knowledge 
utilization In these districts in relation to specific 
innovations; and 

- An ethnographic study of 12 school districts* selected 
froit the larger pool of 145 districts. 
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Initially* five or tfaeifi aubatudlM vak to bfi dfiilgnfid and conducted 
by different research organizations « each operating ao a aubcontractor 
to the prlne' reacarch orginlzatlon-*^lch was also responsible for the 
conduct of the first aubstudy lifted above. Technically* the studies 
appeared to be Independent of each other at the outset. However » for 
the major survey and case study substudles (the last tyo on the shove 
ll8t)« collsbotstlve sctlvltles Incressed over s period of time* Isrgely 
on s managerial basis. The Investlgstors responsible for one substudy 
begsn to exchange findings and tentstlve conclusions tflth the Investl* 
gstors of the other substudy» and this resulted in mutual Insights snd 
sn Intellectual debate about the knowledge utilization process. Although 
no technical synthesis occurred* the final presentstlons of both sub- 
studies were substantively enlightened by these Intersctlons. 

Another wsy of compensstlng for the Isck of technlcsl Integration 
In s psrsllel design is for the same Investlgstors (or principal Investl*- 
gator) to be an active participant In the relevant substudles. Under 
these conditions (fi*g** Bank* In progress; and Sleber* 1974)* the final 
conclusions of th^ master study can reflect evidence from both the survey 
and case study efforts* even though each effort may have been conducted 
on the basis of aeparate technical designs. 

Sequential Designs 

A sequential design atUl falls short of s technically Integrated 

design. However* the case study and survey components are more tightly 

concerned with complementary research objectives* and one of the com- 

■ 

ponents Is deliberately undertaken first* with the major goal of Informing 

^* 
^« 

the design of the other component. « 
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Th« general advantages of using aurveys and case atwUxea In this 
conplcsentary manner have bee^ dlacussed in greater detail elaewhere 
(Slcber, 19^3). As but one possibility, case atudlea can be used to 
reduce *n initially unstructured inquiry into a focused set of measures, 
vhich then can be applied in designing a survey. Alternatively, a survey 
can be used to identify the distribution of a aet of conditions, and 
thereby serve as a 'Wiverse" against vhich a new set of case studies 
can be selected* Of course, a more complex sequential design may involve 
more than a aingle iteration between case studies and aurveys, as a 
string of such iterations may occur as part cf the same investigation* 

The investigations under review revealed no major new lessons along 
these lines* The main finding was that sequential designs were indeed 
among those used to investigate knowledge utillzatlont and that both the 
survey*flrst and case study*flrat sequences could exist* In one Investl* 
gatlon, the initial phase of the investigation Involved an exploratory 
set of case studies (Louis end !>entler, in progress)* One purpose of 
the ca^e studies was to define the iDOst pertinent units of analysis for 
further investigation* For example, although the investigators knew that 
they eventually wanted to analyze the experiences of a set of knowledge 
utilization ^'projects" sponsored by the National Institute of Education, 
the operational definition of these projects lacked the clarity needed to 
design a aurvey of the projects* participants* The case atudlea were 
undertaken to assist in this definitional process, and the aurvey will be 
conducted in a aubsequent phase of the full investigation* 

In another atudy, the initial phase included a aurvey of organiza* 
tions Involved in the agricultural knowledge utilization process 
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(?tlltr tt in progress)* Miny different organizations participate 
In a cooplax unner in this process » with their roles cohering knovledge 
invention* davelopttent» transfer* and utilization. One purpose of the 
organisational survey vas to define these roles and to identify a 
specific set of inter organisational netvorfcs*-dealing with the iaoe 
technology**cach of vhich could later serve as the subject for a case 
study. 1)ius» the later phase of this Investigation will involve the 
design and conduct of ten case studies of Interorgsnlzational networks; 
the definition of these networks would not have been possible without the 
earlier survey work. 

1!he&e two exainples Indicate how sequential designs can be used to 
investigate knowledge utilization. The results of the survey and case 
study components cannot be integrated la any formal » analytic sense » 
but the two components are much more closely related than in parallel 
designs* Managerial integration can also be used to further tighten 
this relationship; with the above two exasiples^ both sets of principal 
Investigators are Intensively involved in both the survey and case study 
phases of their respective Investigations* and this will create further 
interaction between the two phases. In general » the sequential design 
begins to resemble en integrated design^ and the overall Investigation 
appears less as a collection of substudies than does the parallel design. 

Cases*wlthin*Surveys 

Chapter III showed how surveys call for Information to be collected 
from knowledge producers* knowledge users* or both. Furthermore* if the 
survey is based on the prior Identification of a specific knowledge 
utilisation event or medium* the ptodocers and users can be asked to 
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coDDent on the aame experiences* and the investigator can conpare the 
reaponaes frcm both types of parties* 

In either of these situations* the knovladge producers and knowledge 
users MAy be part of the aame "aite**~«.g. » an Interorganizational net* 
vork* a complex organisation auch as a achool district* or even the same 
geographic area. To underatand how and vhy events occur at this type of 
site* a case study can be used. The case study therefore intensifies the 
investigation at one (or more) of the aurvey sites* end in this sense the 
case atudy is designed to exist within the aurvey. 

This type of intensification occurred in one of the studies under 
review (DlHaggio and Useem* 1979). The Investigation dealt with 25 case 
studies* with each case study being defined as an inter organizational 
Connection whereby loarket research knowledge had been produced and used 

(or failed to be used)* in assisting museums* theaters* and other arts 

i 

organizations. Each case study Involved unstructured Interviews with key 
participants*— e.g. * the director of the market research effort and the 
head of the arts organization Intending to use the results. All 25 cases 
were part of a larger pool of market research projects* however* that had 
been the subject of an earlier aurvey« The survey had focused on the 
relationship between the skills of the market research team and the quality 
of the final research effort (end hence would have been classified as an 
R&D Management study according to our definitional criteria In Chapter II)* 
The Investigators felt that the aurvey had not penetrated utilization 
lasues* end therefore designed the follow*up set of case atudies. 

Under these circumstances* the: case atudies can indeed provide evi* 
dene* concerning the knowledge utilisaj^ion process. But because the case 
atudies are part of a larger pool of aites* the results also can be 
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Seneralixad to tuclude ««tinatioD9 concerning thm frequency of occurrence 
of the various c«s« «tudy outcomes* In other vords« When ceve studies are 
nb^ddtd within «urv«y«, the overall invastigation can provide ln£ornation 
regarding both tba proceaa and the incidence of knowledge utilixationr 
Compared with experimental reaearch designs, auch a combination of results 
begins to approach the analytic require&enta for establishing both a 
cauaal pattern of evaota (internal validity) and a framework for knowing 
the statistical generalizability of the results (external validity)* This 
type of mixed design is therefore much more comprehensive and integrated 
than the those previously described (parallel or sequential designs)* 
Furthermore « this type of mixed design is more comprehensive than any of 
the survey<^domlnated or case study'^domlnated designs described in 
Chapters III and IV* UnfortuQately« more examples of this case^vithin* 
survey design were not encountered in our review* 

Surveys*within-Cases 

The opposite design is also possible* In which several cat^e study 

aites are first identified* and widespread survey efforts are undertaken 

at each site* The case studies can still retain their distinctive 

identity under these conditions* because conclusions about a site's 

experiences may be based on several sources of evidence* of which they 

survey results sre only one part* However* if the survey evidence alone 

Is aggregated across sites (a*g** by type of respondent)* and no atten^t 

la made to integrate the evidence within site* a survey-dominated design 

Is actually at work. In aum* the survey s*wi thin-cases mixed design must 

Include some integration of evidence a( the site level (sgain* a **site** 

* 
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can bt u intcrorganizational arraogftmentt a large and conplex orgimiza- 
tlont a geographic ar«a» etc.). 

The aurveya«vltfain-cases design was uaed in three aajor atudies of 
knowledge utilization. One atudy covered 39 conmunity vental health 
centers and their utilization of new knowledge provided by an external 
consultant (Ltraen» in progress). Surveys were conducted of the major 
participants in this process* with reference to specific knowledge 
utilization eventSt ud the Investigation uncovered about 800 inter- 
actions concerning the utilization of apeciflc '"pieces of information." 
In the analysis phase of the Investigation* the aurvey results ^re 
examined across sites* so that comparisons were md^ among a large number 
of knowledge producers (consultants) and knowledge users (staff of the 
community mental health centers). However* the Investigation also 
collected organizational Information about the 39 centers* and this infor- 
mation was integrated vith the survey results from each sire* so that 
39 case studies could in principle have been completed. 

A similar aurveys^i thin-cases design was followed in a atudy of 
educational networks* in which surveys of key personnel were conducted ac 
23 sites (Firestone^ in progress). The analytic efforts also included a 
cross-site aggregation of survey data and a vithin-site integration of 
Infonnation from sources of evidence* not limited to the survev. 

The surveys-wlthin-cases design creates two key problems that must 
be overcome to be successfully Implemented. First* even where the 
Inveatlgation la aggregating the aurvey evidence across aitea (a atep that 
aeems easy to do* given existing techniques of aurvey analysis)* the mixed 
design can lead to an unequal dlatribution of respondents across sites. 
"Larger" aitea art likely to have had more participants interviewed than 

ERIC 64 



58 



***mAller" aitea* and tte reaultiog aggregate evidence atust compensate 
for this diaparity through aaw weighting procedure. Tbe weighting 
procedure* Ui turn* Bay cause additional difficulties not anticipated 
hy the inveatigatora. Second* the aurvey and nonsurvey evidence* within 
aite* vuat be Integrated through aome fonul procedure* otherwise there 
will be Insufficient assurances regarding the potential over- or under- 
reliance on the various aources of evidence. Neither of the two above- 
mentiv4.ed atudies had developed* at the time of our inquiry* fully aatis-* 
factory procedures for dealing with either of these two problems. 

Another study under review was able to address, to a large extent, 
the second problem»of integrating aurvey and nonsurvey evidence within 
aite. In this study, about 90 school districts had been the subject of 
various types of data collection, ranging from surveys of key participants 
to field observations to case histories produced by a "site documentarian'* 
(Louis and Rosenblum* 1981). To integrate this vhole array of evidence, 
the investigators created a meta-research instrument* called a Consoli- 
dated Coding Form. One auch form was to be completed for each of the 
90 sites, by a m^iatber of the Investigating team who had access to all the 
aources of evidence about the site. The Consolidated Coding Form thus 
aerved* in a way* like a case atudy protocol, except that the integration 
of evidence had to rely on data that bad already been collected. The 
Consoli'lated Coding Form then became the basis for reporting both the 
vithin-site findings and cross-site trends. Such an approach may there- 
fore be regarded as but one attempt to deal with the problem of integrating 
aurvey and nonsurvey evidence* vhen^the aurveya-vithin^-cascfs design is 
used. 

'es 



59 



ERIC 



Overall^ ahould these two problems be autiDottntable in the future, 
the Aurveyt-vlthin-cAtes design also offers ao&e advantages over other 
rasearch designs. The surveys can become the basis for assessing the 
frequency of events within a given site, while the cases can be used to 
develop insights into the knowledge utilisation process at each site. 
Across cases, a iDultiple*case replication design (see Chapter IV) can 
atill be usedf mjxd the entire investigation can be used to develop 
analytic rather than statistical generalisability* 

S fflflnary 

Mixed designs are those in which a single investigation relies on 
both the survey and case study strategies* in general, these designs 
offer advantages that neither the survey alone nor the case study alone 
can provide* Mixed designs can therefore be the basis for a more 
thorough investigation of fcnowledfs utilization. 

Several types of mixed designs have been identified and were used in 
the studies under review* The variations cover situations in which there 
is no analytically formal integration of survey and case study evidence 
(the parallel and sequential designs) to situations in which such formal 
integration is possible but difficult (the cases *vithin-*surveys and 
aurvevs-^thin**cases designs)* Our discussion has indicated some of the 
experiences that others have had vith these designs, including the 
problems that they have encountered* Nevertheless* when properly executed, 
the Istter two des^^^as may be far mor%. preferable than the parallel and 
sequential designs* and investigators may want to make greater use of the 
preferable designs in fhe future* 

However* the choice of the ultimate research designs is not always 
under the full control of the research Investigator* In fact, external 
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forces mmy ht mo «troiig that the f laal reaearch design Is highly con- 
strained before the Investigator has an opportunity to explore fully all 
of the possible alternatives. The next chapter deals with some of these 
external forces* 
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VI. WtOJBCT MAMAGIM2HT AHD RESEARCH DESIGN 

In theory^ • rtfturch deiicn should bft cmutructftd ^olftly vn tfaeV 
bMii ot technical tcon«idft»tion«. Th« preceding chapters h«ve described 
sOM of thsie coQSiderstiofis in studtylng kitovlsdge utiliuticm, vbether 
surveys or esse studies have been selected ss the desired rssssrch 
sL^stegy. 

Bovever* nontsclmicsl considerstions slso csn sffect s research 
design. These include issues of: 

- Project stsff skills; 

* Project orgsniistion; 

* Resources for conducting the rcsesrch; end 

- The organltstioD end timing of reporting requirenents 
for comunicsting the resesrch results- 

« 

In uny csBes* these nontechnlcsl considerstions sre directly under the con^ 
trol of the research investigator; but In esses of "procured" or sponsored 
resesrch, an sdditionsl 'at of influential persons includes the staff of 
the R&D-fundlng agency (Yin, 1980). 

Our review of knowledge utilisation atudies L^esled s surprising 
degree to which these nontechnlcsl considerations could sffect resesrch 
'^nifpn, Fttrthervore» few persons appesr to be aware of these relation- 

*v. 1^ically» the nontechnical considerations are viewed froA s 
M^iiagerial perspective alone» on the erroneous sssuiq^tion that aanageoent 
choices can be Made without affecting the technical design of s study* 
The purpose of the present chapter is therefore to identify aome of these 
relationihips, and to Indicate how certain seemiagly unobtrusive choices^ 
whether in designing s Request for Proposals ^F?) or in planning and 
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organising the actual research proJect~can actually have a large Impact 
on the rataarch datigii of a study. Because this type of information has 
not been vell-docuocnted within the formal reports of research investi- 
tations» much of the evidence for this chapter comes from two aupple- 
mentary sources that were ^^xamlned as part of our study: interviews with 
principal investigators and analyses of RFPs or other research solici- 
tations that led to a knowledge utilisation study. 



Project Staff Skll^Ls 

The training* experience* and qualifications of a project staff will 
affect all asper^s of a research project. Not surprisingly* the 
identity of the key Investigators is a key variable in the consideration 
of rei»earch awards* Whether such awards are grants or contracts. 

Where research solicitations attempt to specify the desired qualifi* 
cations of the key project staff* attention is typically given to 
substantive skills. For instance* some RFPs give general descriptions of 
the types of research experiences that might be relevant: 



^ . . To plan* conduct* and complete this study [of federal 
and state dissemination strategies] successfully* offerors 
vill have to provide staff Who* together* have technical 
expertise and knowledge in the areas of diffusion* program 
evaluation* research design* instrument development* field 
data collection* statistics «Bd sampling. Offerors are also 
encouraged to supplement their staffs with persons who have 
done similar work in disciplines outside of education (e.g.* 
sociology* anthropology* social psychology* political 
science* communications)* • • • The proposed project 
director should have expertise in planning and managing 
evaluation studies. (RFP OE-78-100) 



Similarly* unsolicited proposals will «>phj ' ^ the previous experience of 
the proposed Investigators* i& relation, to :opics proposed for further 
investigation. 
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Thl« tBpfaMlf t«nd« to underplay the ■ethodologlcal training of 
the proiK>«ed invettlgetors* Although every investigator nay fa«ve an 
ability to deal with a vide range of research Bethods^ aost investigators 
vill have been trained In academic departioenta with dominant method* 
ologlcal leanings. For Instance » sociologists trained at the University 
of Michigan^ especially with expoaure to research projects at the Institute 
for Social Research^ will tend to be best acquainted with survey methods as 
the primary means of gathering evidence. These methodological proclivities 
are not necessarily related to apeclflc academic disciplines. In addition, 
the proclivities may change over time at the same academic department^ so 
It becomes Important to know when someone studied at a particular depart- 
ment, and not Just where he or she studied. 

This methodological dominsnce will mean that, ceteris paribus, an 
investigator 17III favor d1,^£erent t- ^es of evidence In designing a knowl- 
edge utilization study. As indicated previo^jsly In Chapters III, IV, and 
V, tMs orientation Is therefore likely to lead to the aelectlon of so^e 
designs and not others. Among the studies under review, an expected 
observation was the finding that those trained previously In surveys 
tended to use surveys In their investigations, whereas those trained In 
case studies tended to use case studies. 

The obvious effect of an investigator's methodological orientation 
need not be belsi'^red. The salient point for this report, however* is 
that auch methodological orientations are rarely identified as en explicit 
component In Judging an investlgator*a qualifications for conducting 
knowledge utilization research. Whereas an Investlgstor will therefore 

be selected to participate In a research project on the basis of aub- 

s 

stantlve experiences and knowledge, lif fact the choice will also 
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(uDkjiawlBgly) Affect the reeeerch desigii beceuee the investigator will 
alio have a particular methodological orientation* In thia MnQer* the 
aelaction ot project ataff will affect the research design of the 
ansuing project* 

This same principle nay be extended to the identity of the specific 
rasearcfa organizations and academic departments that are to be the 
grantee institutions (or contractors) for a given atudy* Some Institu* 
tions may have a normative preference for Ci^-rtain kinds of methodological 
approaches^ and the research team nay be responsive to such norms* Many 
organizations and departments* of course* can support the full variety 
of methodological approaches* so that this institutional factor* while 
relevant* probably has only a minor effect on the research design in most 
cases* 

Project Organization 

The traditional research project may be considered a itnified research 
team* usually composed of fever than three or four individuals* working 
collaboratively on the entire project* 

Several of the knowledge utilization efforts under review* however* 
vere sufficiently large that a more differentiated organizational pattern 
was needed* Unaer these conditions* research teams had at least three 
choices* all of which were found aiDong the studies reviewed. The first 
pattern was to emulate the traditional research projact* even though the 
research team might have been composed of five* six* or even vore members* 
This pattern called for intensive Interactions among the members of the 
team* including weekly meetings and'daily working relationships; all 
members shared equally among the variopB research tasks* 
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Tha AAcond pattara vm to decentralize the eteff into eeveral com- 
|KnieDte« vltb verioue eubgroupe^^^g* « tvo or three peraonB^-rtsponsible 
for varioui faceta of tha reaearch project* This pattern allowad for 
AOMe functional and methodological apeciali ration auong fhe facats« and 
«aa Bore likely to satisfy profaasional needs for individual recognition, 
in contrast to tha first pattern (in ^ich no single sember apecializes 
on any givan topic or nethodology) * 

The third pattern was an extension of the second, but involved the 
use of multiple organlsations^-e^g* , subcontractors* Under this pattern, 
the components of the research project were likely to assume the nature 
of "aubfitudies/^ with an independent research objective, design, and data 
collection effort of their own* The original research project could lose 
its own technical coherence and identity, becoming instead an administra-^ 
tive and managerial unit for integrating the aubstudies* 

None of these patterns is necessarily the correct or desired pattern 
for all knowledge utilisation studies* However, each pattern is likely 
to favor a particular research design* For example, the more decentralized 
patterns are clearly likely to result in the use of one of the mixed 
deslgns»the parallel design<^*^en both case studies and aurveys are to be 
conducted* Solicitations that (knowingly or unknowingly) encourage the 
creation of managerial substudies end independent research teams vithin 
the same investigation ara therefore favoring parallel designs* Sinilarly, 
aolicitations that attempt to reduce time gaps by calling for case studies 
to be completed while the project team awaits field clearance for its 
surveys may be Indirectly favoring aaquential designs* 

As another example, one of the* underlying principlea revealed In the 
previous chaptera has been the degree Jto ^Ich a knowledge utilisation 
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Atttdy lntat»^«d inf oiutlcm f ron kiioiil«dge produc«» and uaera* When 
•Itfaar tfa« aecood or third— decantraliMd'— patttru of >roJtct 
ortanlsatlon '«re u«td» the rtault la likely to be « dealfA In uhlch 
diffarant aspects of the knovladge utilization pro^eas are Initially 
axaalnad aeparately* Dlfferant fsfforta vlll he sountad to Investigate 
the role of knwladge uaera» the role of knowledge producera» the role of 
Inteiaedlarlea (If any)» and the type of knowledge being utilized (e«g«» 
Loula and Roaenblum* 1981; Crandall» in progrcas; and Doctors at al«» 
1979)* The project aiy ultinately draw together the lessons from these 
separate components » hut the Inltlsl fr^^gmentstlon can potentially lead 
to Incompatible terminologies and research operations^ making the final 
ayuthesls more difficult* In any case» the disparate efforU are not 
likely to reflect a alngular research design* 

Once agsln» although the nature of a proJact*a organization can 
affect the research design In this manner* the nature of the organization 
la cCten dictated by nontechnical factors and the overall level of 
resources* In addition* It ahould be noted that the type of project 
organization Is not necessarily controlled entirely hy the research team* 
as certain preconditions may have been Implied In the original research 
solicitation or RFP* 

Resources for Cdoductlng the Research 

The overall level ^f resources available to do a study will also 
affect the research <IeslgB» especially If the atudy is auppoaed to cover 
a minimum nuskber of altes or dsta points* The lower the resources in 
relation to the number of altes or data points* the iiore the atudy will 
be limited to aurvey dealgna* Thla Is ^ecauae aurveys can be used to 
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cover the necessary sites or data points, •vcB i£ tbfi cvidfiBCfi ift some* 
tA«t iuptr£ici«l. In extreme eM«if the eurveye caa uee lo%r-coit mailing 
or telephone technique! of date collection. By conparlson, even though 
**alnl*caBe studies** can ba daalgned «t low coit« the travel expenses 
general required to conduct case studies will stIU nean that the survey 
Mthod will yield a higher number of sites or data points. Finally, as 
pointed out In Chapter V, nixed designs are likely to require the 
greatest investment of resources. 

The level of resources available can affect the research design in 
another way. To the extent that the resources dictate the use of less 
experienced research personnel fcr conducting fieldt^ork, any case studies 
are likely to be probleo&atic, and some ty{»e of ourvtty design may be wore 
preferable. Thus, the vholidtlc case study design especially requires the 
use of experienced and aenlor Investigators, irfio often spend conslder^^bl^ 
time at a site (e.g., Alkln et aX., 1979). H6reover> the vholistic design 
may exist as part of a mixed designee. g.^ the caseS'^thin-surveys design. 
In contrast, when a case atudy design is not ioplesnenteiJ by experienced 
field investigators, the pursuit of nev leads in the field as veil as the 
final assembling of the case report may he problematic. 

As a further note, this difference between experienced and lutz 
experiencf.d investigators in collecting field evidence is in ffict « more 
general difference between survey and experlaental Investigations on the 
one hand and case studies on the yther (Vin, 1981b) r In the first tvo 
research strategies^ data collection is best conducted vtien Che investi- 
gator has a minimum of Individual discretlon—e.g. , the ideal survey 
Intervlever vill follow a strict protocol for giving instructions find 
eliciting responses. In the case aturfy strategy* hovever, a fieldvorkcr 
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miMt have a full undtrttanding of the •ubttanee of tbt investigation » 
•0 tltat discretion can Ibe exercised Intelligently and systematically, 
to take tdvnuge of ntv infonutlcm that may arlae during the data . 
collection affort* 

The OraanliatloD and Tlnlng of Hefiorta 

Xn traditional raaaarch projacta^ there say be little correapondence 
between the organltatlon and tlolng of final raporta— typically taking the 
fors of Journal artlclea**and the vay in which the research was organised* 
The sore experienced research inveatlgatora maintain a atream of publi- 
cations that draw from various reaearch projecta^ many of which suiy have 
been coopleted a While ago* To this extent^ the reports clearly follow 
the conduct of the reaearch » and the reports do not affect the reaearch 
deaign« 

Htywever» most knowledge utilisation atudles result In one or sore 
reports about the research^ with aoine reports being produced before the 
reaearch la conpleted* This eophaals Is often the result of s need to 
ahow InterlB progress on s lengthy study » ^r possibly to serve the early 
Information requireaents of * particular audience* Vnder these condi- 
tions » two typea of interim reporta may be prepared* The flrat la a full 
progreaa report » capturing the coaplexlty of a reaearch project and 
touching upon likely trends and patterns » hut In a manner dictated by 
actual aventa (end actual progress)* The second la a truncated progress 
rep>rt» based mainly on the completion of a apeclflc data collection 
effort or the vork of a aubatudy component* In this latter altustlon» 
the reporting requirements may have the effect of fragwnting the reaearch 
dealgn* ^ 

76 



TO 

For liut«nc«» In • study uslnc • alud dMlfD» If teptnte Interim 
nportt ar« r«qulr«d for covering the FMultt of • turvty w4 the results 
of a set of case ttudlM* this 1« aore likely to lead to the "bae of the 
parallel daalfD» ao that the rtaearcb tean has a hlsher chance of com* 
plying vlfh the Interim reporting requlrcmenta^ Even though a later 
raquirement may apecify the Integration of the aiirvey and caae atudy 
«vldence» the Intagrative effort may only be mounted later in the atcdy» 
rather than being an InUgral part of the original deaign^ la ae: 1 
of the knowledge utilization atudiea under reviev« this type ot fiagoented 
reporting structure vss typically required by an R&I>*flndlng agency that 
bad "contracted" for the reatarch^ 

In contrast » Where Investigations vere bssed on the awarding of 
resesrcb granta (e«g«» Larsen» lnprogress)» progress reports Mrs 
frequent but Wholistic^ In addition* the principal Investigstor 
repeatedly used orsl briefings to keep the R^^IHfunding agency apprised 
of the proJect*a progress^ This Integrative reporting Yequirement» 
hovever» meant that the technical ccAierence of the proJect*a overall 
resi^erch design could be maintslned^ All sources of evidence vere used 
to srrive st major conclusicas» tfarouglhout the effwt; and the final 
report vas but the culniuation of this process* and not based on a 
apecial integration of findings occurring st the end of the project 
only« 

Sumary 

This chapter has auggested that four typea of project management 
laaues can affect the reeaarch deai^n of a knowledge utilisation atudy; 
project ataff akills» project organisation^ the reaources svailable^ and 
the organization and timing of reporting requirements^ 
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In wwt ew««» tfa« d«cl«loiii about fh««e wnagerial Mpect« of con- 
dttctlBg r«ie«rch have not hmen wpllcltly linked vltb the potentlel 
effects on e reeeerch deelfo* In fect» project unegesent leeuee ere 

tteuelly decided eccording to e aenegerlel perspective thet ie dleeeeoclated 

* * * 

frm the technical deelgn proceee» end fhe potentlel reletlonehlps between 
project aenegeaent end reieerch design ere not likely to be epprecleted^ 

To geln e fuller Inelght into the poeelble etrength of thle reletlon- 
ehl>» one ney conelder fhe reveree order of declelonmeklng^ Normally* the 
project aenegcaent leeuee ere eettled flret» either beceuie of the con- 
etrelnts of the R&D-fundlng egency*e resourc«:s or beceuse of certeln 
declelons in designing reeeerch eollcltetlons^ However* the research 
desig:n could eltemetlvely be etlpuleted flret. If It were« the project 
aensgeoent fectore would in turn be dlcteted by the research design* but 
would likely entail a vuch wider verlence then R&D-^fundlng egencles or 
reseerch orgenlsetlofns alght tolerete* Beceuse project nenegement Issues 
aust therefore be eettled first in nost csses» the decj^slons do sffect 
the reseerch design* end the purpose of this chepter hes been to suggest 
eoae of the weys In which these effects have occurred in knowledge 
utUlsetloo etudles* 
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EeferencM Chapter VI 

KF? Ofi-78-lOO (full dtttlOQ to be "We^ l«ter) . 

-Th» Cm* Studv M • Serious Research Strategy, *' 
"^:ifj p,'-e,II^li" '<"''^"°tlll«tie.. S.p..^« 1981b. 



3:97-114. 



* » Contro"«i Evaluation? The Inter or ganl«atlonal Com- 

lieSJSs oi*E^«U« R«e«ci." in E. t. Baker (ed^). Ejjlua£4a£ 
e.!i!^L;L./i"?ro.ra^, Center for t»« "udy o ^^uatlon. 
Univeralty of California. U* Angeles* Calif., 1980. 

Other citations In this chapter may be found In Appendix C of this 
report (see pp. 86-88). 
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VII. CONCLUSIONS ABOUT QUALIIATIVE RESEARCH DESIGNS 

Our study has used the knowledge utiUMtion proceta to Illustrate 
the problem of designing qualitative research. Four major aspects of 
research design have been considered: 

- The problem of defining the topic to be studied; 

- Designs that are dominant vhen survey methods are used; 

- Designs that are dominant vhen case study methods are 
used; and 

- Mixed designs « in vhich surveys and case studies are 
used in a complementary manner. 

As a result* our study has identified a number of research designs that 
have been used in 32 previous investigations of knowledge utilization* 
covering a range of urban services and policy applications. 

What was most revealing was that there seem to be a limited number 
of designs. The 32 previous investigations produced a uniform pattern* 
regardless of the specific service being examined. Our guess is that 
future invest igationa may also be limited to these designs and that the 
major contribution of our own study vill have been to begin the codifi- 
cation of qualitative research designs. 

Our study also indicated a few ways in vhich project management 
choices can influence the design of a qualicative research study. In 
particular* project staff skills* project organisation* the level of 
resources* and the nature of reporting requirements can all affect the 
research design of a study* even tfaou^ these management aspects are 
generally considered Independently of the technical design of a 
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Atudy. In Aoae iltuatlons* the project Mnagesent factors vill affect 
irticther eurvtye or caee etudiee ere likely to be chosen as the ttajor 
etretegy for collecting dete; in other eituetions* the project unage- 
■ent factors cu affect the degree of coherence or fragmentat^n in tfie 
overall rcaearch design. 

Theae general findings provide e framework for categorizing and 
analyzing qualitative research deeigns. Such an effort has been rarely 
attaiBpted In the pest* and the etudy*s findings can therefore esslst 
rcseerch investigators and R&D<^funding egencies in designing future 
qualitetive research. As a descriptive effort* the findings hopefully 
serve a useful purpose already. 

The purpose of the present chapter^ however* Is to explore beyond 
the descriptive stage, and to speculate /^bout the potential Implications 
of our findings. In particular* ve would like to consider Aether the 
findings can truly be generalized beyond the topic of knowledge utiliza- 
tion* and ^etlier the findings lead to any suggestions for intervening 
in the design processes atep that assumes some understanding of the 
causal links In the design process. These topics are discussed in the 
remainder of this final chapter. 

The Ceneraligability of the Findings 

The search process described in Chapter II led to the identification 
of numerous empirical studies* each appearing at first to be a study of 
the knowledge utilisation process. After further cxaminat' -i, only about 
half of the atudies actually satisfied our definition of the knowledge 
utilijcation procesB»i.e.* that an ^vestigation had to be concerned 
with the consDunlcatXon of .knowledge from one party (a knowledge producer) 
to ai«other party (a knowledge uiier). The nonqualifying atudies 
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•aphaiiMd aome reltt«d topic~e.g.^ diffusion^ organiutionAl change^ 
UP wuitgeQCDt^ or DetvorkiDg»aiid Chapter II «howed how these related 
topics could he considered one contaxt for knowledge utilisation research. 

Our search process was exhaustive and extensive. Numerous printed 
and personal sources were consulted. Thus^ we believe that the studies 
ultimately identified as knowledge utilisation atudies do represent the 
overwhelming majority of studies on this topic* To this extent^ we 
believe our findings to be fully applicable to the problem of designing 
research on knowledge utilization. However^ the initial purpose of this 
report was to deal with the problem of designing qualitative research in 
general^ and not just to limit our findings to the topic of knowledge 
utilization. To what extent has this broader purpose been served? 

A quick perusal of other qualitative research topiC8»e.g. ^ program 
management^ neighborhood development* family or cultural interactions*** 
suggests that the essential design char act ei'ls tics identified in this 
report are applicable to these other topics as vell^ vl^h one possible 
exception. The potential match is good because both surveys and case, 
studies appear to be the dominant research strategies for studying these 
other topics in qualitative research* and the subcategories of designs 
(e.g. ^ the wholistic case study vs. the embedded case study) seem to 
characterise these other topics as well. Some translation may be needed 
for the survey designs^ where various population groups (depending upon 
the topic being studied) would have to be substituted for our use of 
"knowledge producers" and '"knowledge users." Nevertheless^ the findings 
regarding the advantages of surveyS":*to establish the degree of prevalence 
of a phenomenon and to determine^ under certain conditions^ the per- 
ceptions and attitudes of the how^ participants in a coiq»lex process** 
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Appaar applicable to other qualitative research topics « and c^t Just 
knovladge utiliaation. 

The aingle txception Bay have to do with a third reaearch atrategy 
iot dealing vi4;h qualitative research* but that was not domioant in 
studying the topic of knowledge utilization: the. analysis of docui^eiitary 
and archival records (e.g.« Uebb at al., 1966). These records, such as 
census tract d&ta or other municipal and governmental data files, are 
often directly relevant to the Investigation of a qualitative research 
topic. In some studies, the records nay be the sole source of evidence. 
Slid this has bscone especially true with the develoF&ent of the subfield 
^'quantitative history,** whose techniques are aimed solely at the analyf^is 
of archival recor-ls. However^ these records are not a significant part . 
of the knowledge utilization process, and the pertiv^at research designs 
for using such records therefore ignored in our report. To this 
extent, our rendition of research desig:n8 for qualitative research is 
incomplete, and some topic In which these kinds of documentaty md 
archival records have been prominent ought to be one of the priorities 
for extending the present work. To our knowledge, although some attention 
has been flven to data collection techniques In using archival records 
(e.g., Cochran, Gordon, and Krause, 1980), no attempts have been made to 
determine whether unique research designs are also involved. 

In spite of the void regarding this third research strategy, the 
findings in the present report uy still provide the basis for further 
elaboration of qualitative research designs. Further corroboration con- 
cerning the Inpo^taoce of olxed (survey and case study) designs would be 
especially critical. The vore robust finding, that aucfa mi»d designs 
were the most desirable for other qualitative rescftrch topics besides 
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knowltdge utilization, vould Msl^t in developing e design^counterpart 
to the turvey-cMe ttudy coiq>l«m€nUrity that is already k&own about 
data collection^ 

Thus, for data collection, the existing evidence etrongly suggests 
that survey and case study techniques can play complementary roles in 
defining terms, testing potential inatruaente* and assessing quantita- 
tive and qualitative factors (e«g«, Sieber, 1973)^ For design, our 
tentative findings regarding the ability of aurveys and case studies to 
address different types of research questions may be regarded as a 
coimtertheme to the sore traditional interpretationf io vhich surveys 
and case studies are considered to be competing strategies for addressing 
th^ same r^s^arch questions^ Our claim vould be that any qualitative 
researcn topic will involve questions of prevalence and questions of 
causal linkSf and that the survey is the better technology for addressing 
the former vhile the case study is the better technology for the latter^ 
In this vayf ve view surveys ^nd case studies as complementary, and not 
competing strategies, and this conclusion deserves further corroboration^ 

Ways of Intervening in the Design Process 

Any l&ssons about research designs for qualitative research vould 
not be helpful unless they suggested ways for intervening in the design 
process — and thus to Improve future research designs^ Are such inter- 
ventions possible? 

Clearly, the greatest biarden must lie with the research investigator^ 
If designs ate to be Improved in the future, research Inveetigators must 
first be trained to understand the potential variations, and then know 
hov to aelect the ^OBt feasible and ef^ctive design, given the nature 
of the research questions^ To datef docuntentation about qualitative 
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rMUrch dcilgna bu been epirie^ end invc6tlgetor0 h«ve not been exposed 
to the pOielblXlty tbet euch options might exist In the first piUce. A 
Mjor breskthrough, suggested by the present report. Is slaply that 
qualltstlve rcscsrch uy hsve Its own get of fonoal designs, different 
in nature but slsillsr in purpose to those used In expertmantsl end quasi- 
cxperlAentsl research. However, ouch further corroboration is needed 
before research investigators can have at their disposal a ready catalog 
of designs » similar to those available in experijaental or quasl-^ 
expcrljnental research (e.g., Campbell and Stanley, 1963). 

For this reason, che potency of another aource of intervention should 
also be explored. This aource has to do with the role of R&D*fundlng 
agencies. Our report has already auggested that such agencies play an 
implicit role in designing research on knowledge utilization, as a result 
of the content of research solicitations and lUTPs (see Chapter VI). If 
project management factors can affect research designs, and if It&D*fundlng 
agencies have control over project management f actors, the agencies can 
clearly exart some influence over the desl'^ of qualitative research. 
Because few investigators would be wUl^^g to allow K&D^fundlng agencies 
to design their research directly, the use of project management ^actorB 
as a point of leverage may even be more potent and attractive. 

Qiapter VI has already pointed out the various vays in vhlch project 
management can affect research design. An important further step vould 
be to corroborate these findings with some topic other than knowledge 
utilization, and then to develop aooe general guidelines for X6D-fundlng 
agencies. Such a point of intervention may be more effective, in the 
abort run, in producing improved designs for qualitative research. 
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STUDIES HOT HEETING DEFINITIONAL CRITERU 
(though initUlly mppemring to be knowledge utilization etudies) 
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